Quantitative risk management in project management is the process of converting the impact of risk on the project into numerical terms. This numerical information is frequently used to determine the cost and time contingencies of the project. This paper discusses some of the principles of quantitative risk assessment methods, and how these were developed for use on a capital project in the mining industry. Several methods of contingency determination, which are based on the results of a quantitative risk assessment, are explored. The paper shows how the developed process was applied to a real project, and concludes by highlighting some of the pitfalls with quantitative risk assessments, and how they can be prevented.
Overview
Project risk is defined as “…an uncertain event or condition that, if it occurs, has a positive or negative effect on one or more project objectives such as scope, schedule, cost, and quality” (Project Management Institute, 2013, p. 310).
The aim of project risk management is to identify and minimize the impact that risks have on a project. The challenge with risk management of any kind is that risks are uncertain events. In the management of projects, and the subsequent operations of the project's product, organizations attempt to reduce their exposure to these uncertain events through risk management. This is usually done through a formal management process which consists of the following steps: plan risk management, identify risks, perform qualitative risk analysis, perform quantitative risk analysis, plan risk responses, and control risks (Project Management Institute, 2009).
There is some debate as to the origins of the word risk, but it is commonly accepted that the ancient Greek word “ριζα” (pronounced “riza”) meaning “root, stone, cut of the firm land,” made its way to the Latin word riscus, which means “cliff.” The original Greek word was a metaphor for “difficulty to avoid in the sea,” and ancient mariners, picking their way through the numerous islands in the Mediterranean, Aegean, and Tyrrhenian seas, were quite familiar with the meaning and impact of the word. The word was later borrowed by the Italians as the word rischo and rischio, then by the French as risque, and on to Spanish as riesgo. In the 16th century, the word was adopted by middle-high-German as Rysigo meaning “to dare; to undertake; to hope for economic success.” It is believed that the Anglicized form comes from either the French or Italian words (Handzy, 2012).
Project risk management is a well-defined field of study, and numerous books and papers have been written about it. Risk analysis is broadly split into two areas (i.e., qualitative risk analysis, and quantitative risk analysis). Of these two, qualitative risk analysis is most common, and on many projects, it is the only risk analysis that is done. Quantitative risk assessments (QRAs) on projects are less common, often because insufficient data about the project are available to perform the assessment. In some cases, the effort required to perform the QRA may be too expensive relative to the total project value, and the project team may decide against it.
The purpose of a QRA is to translate the probability and impact of a risk into a measurable quantity. The value or quantum of the risk, in the context of projects, is added to the project cost or time estimate as a contingency value. Project risk quantification, and cost and schedule contingency are, therefore, inseparable. In this paper, a number of aspects of risk quantification are explored.
Quantitative Risk Analysis
Galway (2004) discusses three risk elements that concern project management:
- Schedule – will the project be completed within the planned timeframe?
- Cost – will the project be completed within the allocated budget?
- Performance – will the output from the project satisfy the business and technical goals of the project?
Where possible, these risks should be quantified to enable the project team to develop effective mitigation strategies for the risks, or to include appropriate contingencies in the project estimate.
Contingency Determination
Many ways have been proposed to determine contingency. Below is a list of methods that appear in project management literature:
Heuristic Methods
Heuristic methods use experience-based or expert-based techniques to estimate contingency; these include:
- Percentage of Total Values (Moselhi, 1997);
- Predetermined Guidelines (Hollmann et al., 2012);
- Controlled Interval and Memory (Chapman & Cooper, 1983; Cooper, MacDonald, & Chapman, 1985); and
- Case-based Reasoning Model (Kim, An, & Kang, 2004).
Expected Value Methods
Expected value methods multiply the probability of a risk by the maximum time/cost exposure of the risk to obtain a contingency value; these methods include:
- Method of Moments (Moselhi, 1997); and
- Expected value of individual risks (Mak, Wong, & Picken, 1998).
Probability Distribution Methods
Probability distribution methods base the calculation of contingency on predefined statistical distributions; these include:
- Monte Carlo Simulation (Kwak & Ingall, 2007; Whiteside, 2008); and
- Range Estimating (Curran, 1990; Humphreys et al., 2008).
Mathematical Modeling
Mathematical modeling methods use theoretical mathematical models to determine contingency values. These models typically make use of both linear and non-linear equations, and include:
- Artificial Neural Networks (Günaydın & Doğan, 2004; Kim et al., 2004); and
- Fuzzy Sets (Nieto-Morote & Ruz-Vila, 2011; Paek, Lee, & Ock, 1993).
Interdependency Models
Interdependency models use the logical and resource constrained dependencies between activities to determine contingency; these methods include:
- Influence Diagrams (Diekmann & Featherman, 1998);
- Theory of Constraints (Leach, 2003); and
- Analytical Hierarchy Process (Dey, Tabucanon, & Ogunlana, 1994; An, Kim, & Kang, 2007).
Empirical Methods (Benchmarking)
Empirical methods use historical projects to determine factors that drive risk. These factors are then applied to prospective projects to determine the contingency-based characteristics that are shared with the historical projects; these methods include:
- Regression (Lowe, Emsley, & Harding, 2006; Williams, 2003); and
- Factor Rating (Hollmann, 2012; Trost & Oberlender, 2003).
Case Overview
Early in 2015, the author's company was approached by a South African platinum mining company to perform a QRA on a capital project for the expansion of an existing platinum concentrator plant. The aim of the concentrator expansion project (CEP) was to increase the throughput of the concentrator by 18%. The estimated cost of the project was US$62 million. The QRA had to be done according to a QRA process that was developed in 2014 by the author's company, specifically for the mining company.
A platinum concentrator plant treats platinum-bearing ore through a process of crushing, milling, and floatation. The final product from the concentrator is sent to a smelter, and then to a Base Metal Refinery (BMR) to remove metals such as nickel and copper, followed by a Precious Metals Refinery (PMR) where Platinum Group Metals (PGM) and gold are removed.
The specific plant comprised two parts, namely a wet and a dry section. In the dry section, platinum-bearing ore is received from the mine, and the ore is crushed and milled to the required size. In the wet section, ore mixed with water is treated to produce the concentrate, which is then dried and further processed in a smelter. The implementation of the expansion project required modifications in both the wet and the dry sections.
The QRA had to address the impact of risk on the estimated capital expenses (CAPEX) and the project schedule. The project was awarded to a primary contractor who contracted a number of sub-contractors through an open tender process.
QRA Process
The QRA process that was developed for the company is illustrated in Exhibit 1 and briefly described below.
Exhibit 1: QRA process.
Project Scope of Work
The project scope of work is the starting point for the QRA since it explains what must be done and allows the project team to assess what types of risks the project is exposed to. The CEP scope of work was well-defined. Several technical documents, drawings, and design clarifications were available to develop the cost and time estimates. A detailed project execution plan was also available at the time of starting the QRA.
Work Breakdown Structure (WBS)
The WBS and WBS dictionary are developed from the scope of work and form the basis of the qualitative and quantitative project risk assessments. The CEP WBS contained 236 control accounts. Most of the work was outsourced to sub-contractors, and some sub-contractors had multiple control accounts assigned to them.
CAPEX Estimate
The CAPEX estimate is developed with the WBS as one of its primary inputs. The level of scope detail that is available when the estimate is done determines the method of estimation is determined by. It is often found that there are different levels of accuracy for different work packages in the estimate. The method of estimation and the accuracy level of the estimate should be clearly documented by the estimator, since this information will result in better contingency calculations later, as fewer assumptions will be made.
An independent estimating firm estimated the CAPEX for CEP. Under ideal circumstances, the estimator should have obtained quotes for all the control accounts, but this was not possible due to time constraints from the client. The estimator ended up using three techniques to develop the estimate, and indicated the accuracy ranges of each control account based on his risk assessment of each item. The estimated control accounts were classified into high (-15% to +25%), medium (-10% to +15%), and low (-5% to +5%) risk items. The ranges were mainly based on the method of estimation that was used. The high risk items were estimated based on the expert assessment of a discipline engineer, as no drawings existed for these items. The medium risk items were estimated based on historical information of similar projects, and were typically based on a percentage of the total project capital, or based on a unit rate (e.g., meters of pipe, cubic meters of concrete, etc.). Low risk items were estimated from quotations received from sub-contractors, which were based on detailed design drawings.
Project Schedule
The project schedule should be an accurate reflection of the scope breakdown in the WBS, and should ideally have accuracy ranges for the time and effort estimates, as this simplifies the QRA process. The point estimates in the schedule should also be free from contingency. If a schedule does not have estimate ranges, assumptions must be made at a later stage, which may introduce inaccuracies. The schedule should be accompanied by the basis of schedule document, which includes a description of how the accuracy ranges were determined, and how these ranges were applied to the tasks in the schedule.
The primary contractor developed the CEP schedule based on the time estimates from the primary contractor's engineers, as well as the time estimates received from sub-contractors in their tender responses. The scheduler suggested an estimation accuracy of -5% to +15% for all scheduled activities. This blanket approach was not ideal but, in the absence of better information, was accepted.
Project schedules for large capital projects often run into thousands of lines. Further investigation of the CEP schedule, as well as discussions with the scheduler, showed that it would not be feasible to apply accuracy ranges to each activity due to the differing levels of detail in the schedule. The primary contractor's schedule was typically more detailed than that of the sub-contractors. It was, therefore, decided to identify sub-networks in the schedule and to apply the risk assessment to these sub-networks.
Project Risk Register
The development of a project risk register is part of the risk identification process (Project Management Institute, 2009). During the qualitative risk assessment process, the risks are evaluated in terms of their relative probability and impact. The risk register is an important input to the quantitative risk assessment and brings project-specific risks into the QRA.
A representative from the performing organization developed the risk register for the CEP. There were 25 active risks in the risk register at the time the QRA was done. There was one high risk, seven significant risks, eleven medium risks, and six low risks, classified according to a 5 x 5 risk matrix, which scored each risk's probability and impact on a scale of 1 to 5.
Risk/WBS Mapping and Quantum Analysis
The Risk/WBS mapping process maps the risk register to the WBS. This mapping should be done at the level where the cost estimate is done, which is usually at the control account level. In this process, each WBS control account is evaluated against the risks in the risk register to determine whether the risk will have cost and/or time impact. In addition to the mapping, the impact magnitude (or quantum) of each risk is determined. The impact is either quantified as a specific cost or time increase or decrease, or as a percentage range with a particular distribution. The quantum analysis is then used to quantify the total risk of each control account.
In the analysis of CEP, the risks were mapped to the control accounts in the WBS. It was determined that a number of the risks would have a post-project operational impact, as well as a business case impact.
Uncertainty Range Determination
In this process, the risks that apply to each control account are combined to determine the overall uncertainty range for each control account. This process combines the risk impact from three sources, namely estimation accuracy, project risks, and systemic risks.
Another aspect of range determination is the impact distribution of risks. Probabilistic risk quantification methods rely on the selection of a suitable probability distribution to reflect the way in which the value of an estimated variable is expected to behave in the real world. When a probability distribution is selected, an assumption must be made about the behavior of the variable. It is unlikely that the selected distribution will be an exact fit for the variable, but in most cases an approximation of the distribution is sufficient.
Two broad categories of distributions were identified (i.e., distributions that reflect human decision making, and distributions based on phenomena such as economic factors, weather, fluctuations in natural resources, etc.). Factors influenced by human decisions, such as duration estimates, seldom have linear probability distributions. The PERT, beta, exponential, and lognormal distributions are good approximations for many types of human behavior.
Factors influenced by non-human phenomena, such as price changes or production line delays, often have linear or discrete distributions. The following principles were used to select distributions for control account uncertainties:
Continuous Distributions
- PERT, triangular, and double-triangular distributions are used where durations and costs are estimated by a person (usually an expert in their field), and where small incremental changes are possible (e.g., the time taken to paint a wall, or the cost of hourly labor to perform a particular task).
- Lognormal, exponential, or Pareto distributions are used when an estimate can only change to one side. For example, the labor cost for a particular activity may be US$5,000. The industry has seen a number of wage strikes, which have increased wage costs above inflation, hence the risk exists that a strike may occur in the foreseeable future, which may increase the labor cost by more than inflation. The probability that the labor cost will go down is excluded from the distribution, since it has never been seen before. The distribution to model this situation should only allow for the option of an increase (Whiteside, 2008).
Discrete Distributions
- Discrete distributions are used where the cost of an activity, or the time to perform the activity, jumps between specific values (e.g., the cost of a pump is US$1,000, however, there is a risk that the chosen pump may not be able to perform as required under extreme rain conditions). The alternative is a pump that costs US$2,000, which can withstand extreme rain. From this example, it is clear that a continuous distribution cannot be used since there are only two values in the risk distribution (i.e., US$1,000 or US$2,000).
There are, of course, instances where the project team understands the underlying factors of the risk impact, and can choose a different distribution.
Simulation
A Monte Carlo simulation is performed to create a distribution based on the estimates and the defined accuracy ranges. The simulation is done for both the project cost estimates and the project schedule. The result of the Monte Carlo simulation produces a normal distribution, irrespective of what the distributions of the individual estimates were (Kwak & Ingall, 2007). This is known as the central limit theorem, and allows one to determine the cost and time estimates at various probability levels with relative ease.
For the CEP, the @Risk software package was used to perform both the cost and schedule simulations. The suggested contingency for the cost and time was at the P80 level. Given a normal distribution, the P80 level is the 80% probability point on a distribution (i.e., a randomly simulated cost or schedule value for the particular project will be less than or equal to the P80 value, 80% of the time).
Results Analysis
The post-simulation analysis of the results is an important step in the process since it allows all the stakeholders to review and evaluate the results. In this process, the project stakeholders also have the opportunity to verify the results against their own experiences on past projects (Galway, 2004). Significant deviations from expected results can be further investigated, and the input ranges can be verified.
The analysis of the QRA results on the CEP project led to much discussion, since the stakeholders traditionally expected higher contingency values. None of the stakeholders could produce evidence to support their higher estimates, and it turned out that the expectation of a higher value was mainly based on gut feel. The results of the simulation were accepted without modification.
Contingency Determination
The the project manager and the project sponsor determines the final contingency. The final contingency is often, not simply the value from the Monte Carlo simulation, and contains additional costs that may be required by the organization, such as management overhead, insurance, contribution to portfolio management reserves, and so forth. (Vose, 2008).
The CEP project accepted the P80 value for the cost and schedule as the base contingency value. A small percentage of the point estimate was added to the contingency to cater for business case risks that are not brought into the project estimate.
Calculating the cost of a schedule delay for the wet section proved to be a challenge, as there was a predefined schedule of shutdowns, which the project work had to be done in. If the work could not be completed in a specific shutdown, it would be impossible to extend the shutdown, and work must be stopped and delayed until the next shutdown, which would usually be three to four weeks later. The additional cost that would be incurred, if the work extended beyond the planned project duration into additional shutdowns, would not only be the cost of the work during the shutdown, but also the cost of the contractor to have their equipment on site in the period between the two shuts. The cost estimate of an additional shutdown would, therefore, be the daily cost of the contractor between the shutdowns, plus the cost of the work during the shutdown. The contractor's cost between shutdowns on CEP was about one-third of the daily cost of the shutdown. From past projects in the wet section of the plant, it was determined that there was, on average, one missed shutdown in every six months, and a contingency allowance of an additional three shutdowns was made.
Evaluate the Business Case
Once the contingency values are determined, the project business case should be re-evaluated to determine whether the project is still a viable option. If the project is part of a larger portfolio of projects, the contingency may make it a less attractive option in comparison to the other components in the portfolio. The CEP remained a highly profitable project with the recommended contingency included.
Final Results
The final results for the project QRA are presented in Exhibits 2 and 3 below:
Exhibit 2: Cost contingency results.
Exhibit 3: Schedule contingency results.
Discussion
In the development of the QRA process and its subsequent use on the CEP, workarounds had to be found for the number of challenges that the project team was faced with.
The P-value Problem
The determination of an appropriate P-value is often problematic, as many organizations fix contingency at some P-value, usually without a good explanation. The challenge with this approach is that the P-value gives the cost or time at a particular probability, but it does not really aid decision making about the project, since the risk that remains after allocating contingency at a particular P-value is still unknown. For the CEP, the remaining risk was reported alongside the P80 value. Since the normal distribution has infinite tails, the P99.99 value was reported as the maximum risk value. When presented with the point estimate, the P80 value, and the P99.99 value, the decision makers knew how much risk was provided for in terms of additional cost and time, but the difference between the P80 and P99.99 values shows how much risk has not been provided for. See Exhibit 4.
Exhibit 4: P-value and remaining risk.
Three Types of Risk
The total risk that affects the project cost and time is a combination of three types of risk. Project risk is captured in the risk register, and only applies to a particular project. Estimation accuracy risk reflects the uncertainty of the accuracy of the estimate, and is related to the level of detail about the project scope, the method used to estimate the quantity of work or material, and the method used to determine the price. Systemic risks are applicable to all the projects in a particular environment, such as availability of resources, political influences, use of technology, and so forth. It is important to note that the total risk quantity is the sum of the three risk types, for example:
A particular control account has a point estimate of US$10,000, with an accuracy range of ±10%. But the control account also has a project risk that a particular resource may not be available. If this risk occurs, the cost will rise by up to US$1,000 with a triangular distribution. There is also a systemic risk that new technology will be used on the project, which may lead to time delays due to rework. It was estimated that this could lead to a cost increase of up to 15%, but could also lead to a saving of 10% since using the new technology may result in the work being completed faster.
The total risk for this control account would be the sum of the impacts of these risks, since each risk is an independent event and could have an impact on the cost of the control account, whether the other risks occur or not.
Detailed Schedules
One would expect that a detailed project schedule would be ideal for performing a QRA, but it is often quite the contrary. Experience with detailed project schedules have shown that estimation accuracy is often overstated when detailed activities are defined since people estimate work in discrete units, such as hours days, weeks, and so forth. A task that takes three days to perform may be estimated as two tasks of two days each instead of two tasks of 1.5 days each, simply because the estimator is used to working in units of days.
To avoid this problem, a sub-network approach was adopted for schedule QRAs. With this approach, critical chain methods (Leach, 2003) are used to determine the sub-networks in the schedule, and contingency is calculated for the sub-network and added as buffer at the end of the network.
Contingency Burn-Down
Experience has shown that most organizations assign a single contingency value to a project for the full duration of the project. This locks in large amounts of capital for long periods of time. The nature of risks on projects is such that the number of risks should diminish as the project is executed, since the remaining scope of work diminishes. A process of time-phasing contingency over the life of the project was, therefore, adopted. This allows the project to release contingency funds back to the business as the project progresses.
QRA per Phase
The method used for the QRA should match the phase of the project. In early project phases, such as feasibility and concept studies, it may be more appropriate to use empirical models, however, these models should not be used once the project has a detailed WBS and estimate (Humphreys et al., 2008).
Conclusion
When used correctly, QRAs have the potential to add tremendous value to projects. The most significant lesson learned from the CEP project is that projects should be set up for QRA. When this is done correctly, the WBS, control accounts, schedule, cost estimates, and risk register should be designed in a way that makes it easy to determine where risks could impact the project, as well as to quantify that impact. Performing a QRA on a project that is not set up correctly leads to many assumptions about the impact of risk, and the resulting contingency values are difficult to defend.
