Risk management

Abstract

Monte Carlo simulation is a practical tool used in determining contingency and can facilitate more effective management of cost estimate uncertainties. This paper lays out the process for effectively developing the model for Monte Carlo simulations and reveals some of the intricacies needing special consideration. This paper starts with a discussion on the importance of continuous risk management practice and leads right into the why and how a Monte Carlo simulation is used to establish contingency. Given the right Monte Carlo simulation tools and skills, any size project can take advantage of the advancements of information availability and technology to yield powerful results.

Introduction

Projects are often initiated with the intent to solve a problem, such as the need to generate more revenue or reduce costs, the need to maintain or expand existing systems or infrastructure, or the need to comply with new or revised regulations. A project might also be started to find a cure for a disease, solve a socioeconomic issue, improve or implement organizational change, or an assortment of other problems. Projects are generally the manifestations of the solution to a problem, such as the development of oil or natural gas as a result of a recent discovery, expansion of the company's computer network to support growth, development of a new product to support an emerging market, construction of a highway, laying a new interstate pipeline, erecting a new hotel and casino, and many other kinds of projects. All projects, regardless of the problem they intend to solve or their scope, have an initiator, sponsor, or client. The sponsor establishes the project's purpose and its objectives. Most projects have specific objectives regarding the cost and timeline in which the project is to be executed. Indeed, there are a multitude of uncertain events or risks that can occur throughout a project's life cycle to enhance or threaten the project's ability to satisfy objectives. These events are identified and managed as part of a project risk management process, yet many projects are allowed to get too far down the road before the risk management process is engaged.

Projects have a cost objective. Projects have multiple phases or stages and, toward the end of each phase, a decision is made to fund the next phase based on the project's progress against its objectives and other criteria that the sponsor identifies at the beginning of that phase. Typically, a cost estimate is prepared for the entire project in one of the early phases of the project and then refined in a subsequent phase. For large complex projects, cost estimating can be quite complex and, in the early project phases when little has been defined, large amounts of uncertainty can exist. Early project estimates may have probable outcomes, ranging from +/- 50% of the estimate, and be referred to as a rough order of magnitude estimate. To account for some of this uncertainty, the sponsor often requires an amount set aside for contingency. Contingency reserves are used to pay for variations in the estimate, the cost impacts of risk events, and the cost of risk response, not for scope creep. Real questions emerge: “Where does the +/- amount come from?” “How is the amount calculated?” Monte Carlo simulation is one of numerous methods available to estimate contingency. People associated with projects avoid Monte Carlo simulation like the plague. Their perception is that a tremendous amount of work is needed to prepare for this simulation. Today, given the software tools available to us, Monte Carlo simulation is significantly easier than in the past. There are a number of hazards to consider and much care should be given when using a Monte Carlo simulation. Use of correct expertise in the Monte Carlo simulation method facilitates successful contingency planning. Although contingency planning is important, complementing the effort with skillful use of Monte Carlo simulation can yield powerful results.

Foundation in Risk Management

Good project managers will naturally desire to achieve their project's objectives; hence, good project managers will take the time to develop a well-considered project management plan. Great project managers will include risk management in their project planning and ensure the execution of the project's risk management plan throughout the entire project life cycle. They work diligently to protect their project's objectives from potential events or issues that may threaten the project's probability of success. They look for opportunities to improve the completion of their project's objectives.

In a nutshell, risk management is a continuous, forward-looking process that is an important part of business and technical management processes. Risk management should address issues that could endanger achievement of critical objectives. A continuous risk management approach is applied to effectively anticipate and mitigate the risks that have critical impact on the project. (CMMI Product Team, 2002, p. 302)

Summarized by Cooper, Dale & Grey, Stephen & Raymond, Geoff & Walker, Phil in Project Risk Management Guidelines: Managing Risk in Large Projects and Complex Procurements, (2005, Chapter 1), the need for risk management is to ensure that:

• All significant risks to the success of the project are identified;
• Identified risks are understood, with both the range of potential consequences they represent and the likelihood of values in that range being determined as far as is necessary for decision-making;
• Assessment is undertaken of individual risks relative to the other risks to support priority setting and resource allocation;
• Strategies for treating the risks take account of opportunities to address more than one risk;
• The process itself and the risk treatment strategies are implemented cost-effectively.

Cost-effective implementation is the tricky part. Risk management is the process of installing a safety net. A trapeze artist perfects his or her routine in an attempt to not have to rely on the safety net. It is not cheap to buy and install a safety net. It takes time to set it up for each performance. Over time, as the trapeze artist becomes more and more rehearsed, maintaining the safety net may soon seem nonessential. Eventually, corners are cut, not literally, but with each performance, one or two supporting members of the safety net's substructure are left out. Less time is spent to ensure that the net is in good working order. At some point, the safety net may just be plain unsafe. Just like the installation and maintenance of a safety net, risk management should be a continuous process that begins at the very beginning or birth of a project.

In Cooper, Dale, et al. (2005, Chapter 1), good project risk management within an organization has the following characteristics:

• project risk management activities commence at the initiation of the project, risk management plans are developed, and risk management continues throughout the project life cycle;
• project risk management is not a discrete stand-alone process, but is integrated with other project management functions; and
• the implementation of project risk management is the responsibility of all project stakeholders and they actively participate in the process.

In Dr. Damodaran's Risk Management: A Corporate Governance Manual (nd, p. 3), he notes that to manage risk we must first define risk. Risk is defined as an uncertain event or condition that, if it occurs, has a positive or negative effect on a project's objectives. (PMI, 2008, p. 446)

Managing an uncertain event is difficult, to say the least. How much money should be spent to manage each risk event? Which risk events are more important? The probability of a risk event is often based on a collection of input from project team members. The impact, generally measured in terms of cost and schedule, can have a range of values, depending on a multitude of variables. The risk assessment process is an important step where, qualitative valuation occurs in order to prioritize risk so the project manager knows where to allocate funds and resources for risk treatments or response plans.

Consider a project with an activity to mobilize a construction management team from the United States to a site in Saudi Arabia (KSA). Numerous risks could be identified around the effect of customs on the shipment of personal computers and other materials needed for the office setup. Generally, a project team member provides input based on his or her previous experience. What happens when the input sounds something like this?

“I have seen KSA customs take as little as one week and as much as 8 weeks. It really depends on when the shipment occurs in relation to Ramadan, which is typically in August.”

In this situation, the project team at a minimum has to consider the following:

• Determine what could cause a delay within KSA customs.
• Decide how to mitigate the potential causes.
• Evaluate whether or not the shipment could occur in the month of August.
• Determine if any of the driving predecessor activities have correlated risks.
• Identify and manage the risks that may impact these preceding activities.

Some projects can have 50 or more similarly complicated risks that need similar consideration. The complexity of the exercise can be daunting. It is similar to herding cats or trying to hold on to the fish when pulling the hook out.

In a project's earliest phases, details needed to assess risks can be difficult to ascertain, as intricacies of the project scope may not yet be fully defined. Getting answers can be just downright impossible. The temptation is often to postpone risk management activities until more scope definition has occurred. The sponsor or upper management, however, wants as accurate an estimate as possible and has little tolerance for ambiguity. This pressure is felt by each functional area on the project. When detailed bottom-up or parametric estimating is required, project contributors will often attempt to protect themselves with padded line item estimates, which can easily lead to an inflated project estimate as well as baseless contingencies, depending on how contingency is calculated.

According to Hollmann (2007, ¶2), the industry can agree on the following four methods to estimate contingency:

• Expert judgment
• Predetermined guidelines (with varying degrees of judgment and empiricism used)
• Monte Carlo or other simulation analysis (primarily risk analysis judgment incorporated in a simulation)
• Parametric Modeling (empirically based algorithm, usually derived through regression analysis, with varying degrees of judgment used)

Monte Carlo Simulation

Monte Carlo simulation is one method used to analyze uncertainty. A mathematician involved with developing nuclear fission technology, Stanislaw (Stan) Ulam, presented a random number sampling idea to address a neutron diffusion problem to John Von Neumann who later presented the idea in a letter to Robert Richtmyer, in March of 1947. (Eckhardt, 1987, p. 132) In 1945, the ENIAC computer had just come online. (Metropolis, 1987, p. 125) It was the first computer to run Monte Carlo simulations, albeit, for a different application than analyzing the uncertainty related to risk events and line item cost estimates. Until recently, desktop computers have not had the horsepower to handle the complexity associated with a properly built Monte Carlo simulation for large complex projects without spending a significant amount of computing time.

Monte Carlo simulation provides projects with a way to handle uncertainty. Projects in the early phases have to balance the demands of accuracy with a scarcity of details. Ultimately, upper management wants to know how much the project will cost and how much funding will be requested. The cost of funding a project is an important aspect that should not be ignored. The capital the organization uses to fund projects, and specifically to support contingency or management reserve, is a tied-up commitment once the approval for project funding is received. The cost of capital used to support contingency and management reserve is a burden the organization bears until the project is complete and the remaining funds are returned. Longer projects should consider a strategically time-based allocation of contingency funds to reduce the cost of capital.

Monte Carlo simulation, if modeled and run properly, will provide cost justification for risk treatments or response plans and a clear and adequate basis for project contingency as well as management reserve.

Project Set Up for Monte Carlo Simulation

Assumptions

• Continuously managing risks and actively managing contingency reserve will reduce an organization's cost of capital funding of the project.
• Monte Carlo simulations require some basic elements when being set up. Both cost estimates and activity duration estimates need consideration of worst-case and best-case scenarios. It is not enough to consider only the most likely value.
• The rule “Garbage In, Garbage Out” must be understood by the project manager and the project sponsor. A Monte Carlo simulation is not a miracle working tool. It is only as good as the quality of information and effort put into it.

Participants

All stakeholders can in some way participate in the risk management process and the Monte Carlo simulation.

Monte Carlo Simulation Preparation and Execution Steps

When the sponsoring organization, decides that a project is going to use Monte Carlo simulation for establishing contingency, then these steps should be followed.

1. Stakeholder Awareness
2. Continuous Risk Management

   a.    Plan

   b.    Identify

   c.    Assess, Score, and Prioritize

   d.    Analyze

   e.    Treatment or Response Planning

   f.    Monitor and Control

3. Initial Estimate Preparation
4. Determine Correlations

   a.    Between Estimate Line Items

   b.    Between Risk Drivers

   c.    Between Line Items and Risks

5. Build Model

   a.    Baseline Model

   b.    Risk Driver Model without Mitigation Plans (Pre-Mitigated)

   c.    Risk Driver Model with Mitigation Plans (Post-Mitigated)

6. Run Monte Carlo Simulations

   a.    Establish baseline

   b.    Pre-Mitigated

   c.    Post-Mitigated

7. Produce and Communicate Results

It is important to recognize that these steps are not necessarily sequential and some may occur in parallel. These steps overlap project controls, schedule development, cost estimation, risk management, and communications processes and are not intended to replace these practices, but rather serve as a guide in preparing to use Monte Carlo simulation as a tool to estimate contingency. Details of the steps are described below.

Stakeholder Awareness

Stakeholder awareness is the act of informing stakeholders, particularly project team members, vendors, and subcontractors, that the project will be running Monte Carlo simulations as part of the project's risk management process. Project team members should be prepared to provide best case, worst case, and most likely values when estimating costs or activity durations for the project. A risk management professional with experience in running Monte Carlo simulations should be included in the project staffing plan.

Continuous Risk Management

Plan, Identify, Assess, Score, and Prioritize

The risk management process needs to start at the very beginning of the project. A risk management plan needs to be written and all parameters for the risk register need to be defined, including how risks will be qualitatively assessed. The full extent of risk management best practice should be deployed on the project. Monte Carlo simulations are part of an advanced risk management approach and are dependent on the other aspects of the risk management process. Projects that do the bare minimum, that is, hold a risk workshop, identify risks, and assess probabilities and impacts to determine risk scores, prioritize, assign owners, identify some mitigation plans, and populate, but not maintain the risk register, jeopardize the effectiveness of Monte Carlo simulations and the potential value added to the project.

Analyze, Treatment, or Response Planning

Each priority risk will have an impact assessed that needs to be broken down and understood by the assigned risk owner. As part of this assessment, the expected monetary value should be determined. The risk owner needs to investigate what type of impact the risk will have, should it occur, uniform or variable. A risk with a uniform impact will always have an impact of the assessed value. A risk with a variable impact will have a worst case value, a best case value, and a most likely value. It helps to understand the probability distribution that most closely reflects the anticipated impact. Probability distribution is discussed in more detail in the Monte Carlo Simulation Tools section.

It is important to properly analyze the effect of risk treatments or response plans on priority risks (risks that are determined to be worthy of the execution of their risk treatments or response plans) in order to determine the residual probability and impact. Again, the impact needs to be evaluated to determine what probability distribution best suits the risk. The residual risk score, which is anticipated after completion of the risk response plan or treatment, needs to be determined and recorded in the risk register.

Monitor and Control

Constant monitoring is imperative to successfully managing contingency reserves. As risk events occur and contingency reserves are drawn down, risk reviews are performed to determine effectiveness of the overall risk management process. This is important for the organization's historical records, which benefit future projects and future Monte Carlo simulations.

Initial Estimate Preparation

As the project defines scope and prepares cost estimates for the various work packages, it is important to capture the details behind the information gathered. Estimates will need to include the worst case, best case, and most likely values and are often referred to as three-point estimates. As procurement activities take place on the project, it is helpful to remember that in the early phases of a project, vendor quotations can be budgetary in nature and not necessarily definitive. Using budgetary quotations for most likely values can artificially inflate the project's contingency calculation. This is typically the most tedious part of the process of preparing for a Monte Carlo simulation.

Determine Correlations

Correlations can exist between any two or more items in the cost estimate, between risk events, and between risk event and cost estimate items. It is important to have a cross discipline review of all correlations to ensure that all functional team leads agree with the correlations proposed for the Monte Carlo simulation. A matrix of risk events and cost elements similar to those shown in Exhibit 1 will help facilitate the documentation of correlations. A correlation is a representation of the dependency. For a chemical plant project, a correlation would exist between steel and concrete. If more steel is added to the job, then more concrete is needed. If for every US$1.00 of steel added to the job, there is an addition of US$0.20 of concrete, then a +0.20 correlation between those cost elements exist. Some cost items may be administrative in nature and increase with an increase to any other cost item. In Exhibit 1, Cost Element #6 demonstrates a correlation of 2% for all cost elements.

For schedules, the correlation relationship is between risk events and schedule activities. Correlations are measured from a range of 0 to 1. Positive correlations relate a dependency that as one increases the other also increases. Negative correlations relate a dependency that as one increases the other decreases.

Exhibit 1 – Sample Correlations Matrix

Build Model

The Monte Carlo simulation models should be structured to be able to demonstrate both the value of the risk treatments or response plans and the amount needed for contingency reserve. Three models are proposed and should be saved as separate files in the Monte Carlo simulation software. It is important to carefully review the project assumptions and constraints at this point to ensure appropriate consideration is given when constructing the model.

Baseline

The baseline should only have cost elements with worst case, best case, and most likely values and correlations between cost estimate items. The worst case values are often referred to as “pessimistic” values. The best case values are “optimistic” values. Most likely, values are either termed “realistic” or “most likely.” Risk events are not entered into the baseline model.

Risk Driver Model without Mitigation Plans (Pre-Mitigated)

Creating a model with risk events and incorporating the full impact of cost and schedule impact areas will help management understand the importance of spending the money needed to implement risk treatments or response plans. The Mitigation Plan term is typically used, because this is the most common treatment or response.

Risk Driver Model with Mitigation Plans (Post-Mitigated)

The final model has all risk events incorporated with all the effects of risk treatments or response plans incorporated.

Run Monte Carlo Simulations

The Monte Carlo simulation software will have a number of settings that the software manual or help files will explain but generally default settings are designed to provide the best outcome. There are a couple areas to be concerned with:

• The number of simulation iterations to run; typically, 1000 iterations will provide what a project needs.
• What probability does management want to use to determine contingency? Generally, P80 or 80% is the total project value to use for setting the final total project cost. P90 or 90% is often used to include the management reserve. Most Monte Carlo simulation software allows the user to configure the graphs and charts to indicate the P80 or P90 amounts.

Produce and Communicate Results

The results are the most important step of the entire process. The results must be well understood by the individual presenting the message. On the surface, it is easy to determine the contingency is the difference between the Post-Mitigation Monte Carlo simulation and the baseline Monte Carlo simulation at the probability level chosen by the sponsor or client. There are often other messages in the results as well. Things to look out for include:

• Distribution is very tight or narrow – may be natural for a project in the later phases to have a tight distribution, as the window for risk events to impact the project diminishes. On projects in earlier phases, this may be cause for concern. Action: Need to review the input data and validate.
• The deterministic value is well below the P20 or 20% mark on the baseline Monte Carlo simulation. This means your estimate is very aggressive and possibly a poor representation of the project's true baseline. This is risky, because the contingency amount may appear unreasonably high and get reduced by management. Action: Validate the most likely values for the cost estimate items with the functional team leads.

The results shown in Exhibit 2 are from a risk review exercise for a large project in which three Monte Carlo simulations were performed.

Exhibit 2 – Sample Results from Monte Carlo Simulation

The Monte Carlo simulation results for this project reflect a tight budget, which was indeed the case. This project also had a large amount (US$400 million) of construction scope, which had been subcontracted on a fixed price contract, thus the construction scope of work risk was on the construction company instead of the project. The red line demonstrates that the project will likely cost 20% to 30% more without spending the effort to treat or respond to risks. The green line represents the possible outcome after implementing the risk treatments or response plans. The bottom line is US$169 million or 14.46% was the recommended contingency amount (the P90 level was used for this project) and risk management will likely save US$150 to US$180 million, or roughly 145 to 15% for this project.

Monte Carlo Simulation Tools

There are a number of tools necessary to properly setting up a Monte Carlo simulation and properly interpreting the results. When setting up the Monte Carlo simulation, items require a stochastic model, where a probabilistic distribution is produced based on the worst case, best case, and most likely values. There are several probabilistic distribution models. The most common are Triangular, Beta, Normal, and Uniform. Typically, the software tool being used to run the simulation will allow the user to select the probabilistic distribution model for each item.

Probabilistic Distribution Models

The Uniform distribution would be applied when there is no central tendency around a mean value. There may indeed be some cases where the cost is estimated equally likely between the worst case or pessimistic and best case or optimistic values. However, if there is a central tendency and the likelihood of pessimistic or optimistic outcomes is about equal, then the Normal distribution would be appropriate. The Beta and Triangular distributions are applicable when the risk is asymmetrical. (Goodpasture, 2004, Chapter 6)

The Uniform distribution is used when the item cannot change and, thus, the worst case, best case, and most likely values are the same. The Beta distribution allows for broader consideration around the most likely values but is more difficult to configure. It is used when there is a high degree of confidence in the most likely value. The Triangular distribution is the more conservative distribution and is recommended if little else is known beside the worst case, best case, and most likely values. Exhibit 3 is a graphical representation of each of the three distribution types with a worst case value of 30, best case value of 3, and most likely value of 8.

Exhibit 3 – Uniform, Beta, and Triangular Distributions

Tornado or Sensitivity Analysis

Tornado or Sensitivity Analysis is critical in identifying the primary risk drivers or events that are having the most influence on the project. This is a useful tool to assist the project in prioritizing which risk event to focus on. Most Monte Carlo simulation software includes this feature for both risk events and cost items. Exhibit 4 is a representation of this diagram.

Exhibit 4 – Tornado or Sensitivity Analysis Diagram

Monte Carlo Simulation Software

There are numerous software solutions available to model Monte Carlo simulations. @Risk® is an add-in for Microsoft® Office Excel, which mostly handles cost risks and risk drivers. ORACLE® Primavera Risk Analysis is a standalone application that will model risks, costs, and schedule.

Risk Management Professional

The process of managing risk and simulating the impact of uncertainty is a complex activity and modeling Monte Carlo simulations is not for the typical project manager, but rather a seasoned Risk Management Professional. Recently, Project Management Institute released a certification program specifically for this capability; not all projects can afford a fulltime Risk Management Professional. Smaller projects still need continuous risk management but Monte Carlo simulations should be farmed out to a professional. The project may need to tap into the organization's project management office or hire a third-party consultant.

Summary

The key to successful contingency planning based on Monte Carlo simulations lies in whether the project manages risk continuously versus a discrete or even a periodic risk management approach. In the past, document and information management challenged, even taxed, projects in ways that made disciplined, continuous risk management too costly. The investment of time and resources to build the model for a Monte Carlo simulation was prohibitive for medium and small projects; only large projects could afford the overhead for such an undertaking.

Today, with electronic information storage and transfer within the project as well as outside the project, capturing the data needed to build and simulate project risks and cost estimates via a Monte Carlo simulation is considerably less costly and less difficult than before. With sophisticated software tools that operate on a desktop, rather than a multi-floor, computer, Monte Carlo simulations run faster and provide a more comprehensive suite of analysis tools. Now that Monte Carlo simulations are easier and faster to prepare, the benefit of this analysis is more readily available for medium-sized and smaller projects.

Managing uncertainty, incorporating contingency based on risk drivers with consideration of cost and risk correlations, demonstrating the cost benefit of risk management and reducing cost capital are easily achievable. Care must be given when running Monte Carlo simulations. There are many ways to create difficulties when it comes to this complex process. Although benefits are real and important, only the skillful use of Monte Carlo simulation can yield powerful results.