Open and closed
the Monte Carlo model
BY ROGER GRAVES
Projects that overrun their budgets and deliver late are common enough. Whatever the causes, one thing is certain—the budget and schedule allowed were insufficient for the work as performed. While better results may be obtained in some instances by improved project management techniques, in many cases overruns occur simply because the original budget and schedule were inadequate. More often than not, this inadequacy is the result of uncertainty surrounding the scope and extent of the work.
When faced with uncertainty in estimating time or cost, many project managers simply assign a more-or-less arbitrary value and hope for the best. However, there is a better way of doing things, based on an understanding of the nature of uncertainty.
Monte Carlo simulation, which models the uncertainty associated with individual tasks, is fast becoming a standard feature in project managers’ toolkits. This simulation can be used to predict overall project costs or completion dates in much the same way that election results can be predicted by polling. Rather than asking every single voter how they intend to vote, the pollster asks a randomly selected sample of voters and predicts the outcome based on their responses.
Monte Carlo does the same thing by looking at a randomly selected sample of possible project outcomes, but takes the process a step further by including a description, or model, of the uncertainty associated with each task. This is analogous to a pollster including descriptions of respondents which summarize the likelihood that they will vote differently on Election Day.
Monte Carlo models uncertainty in projects by assigning a range of values rather than a fixed cost or duration to each task. However, if the range of values is not realistic, expect misleading results.
Open and Closed Distributions
Most Monte Carlo project simulations specify absolute upper limits to the expected range of cost or time for any task. However, setting an upper limit where there is significant uncertainty is merely an act of faith. Exhibit 1 shows a typical model of a task duration used in Monte Carlo simulation. It shows a task taking a minimum of six days, most likely lasting 10 days and having a maximum duration of 15 days. But, what this model also shows is zero possibility that this task will take more than 15 days. Only a foolhardy project manager would say this if real uncertainty was involved.
Exhibit 1 is an example of a closed distribution, or a description of the range of cost or time to complete a task in which any possibility of exceeding the set limits is specifically denied. Descriptions such as this are acceptable where the level of uncertainty is low, but can be highly misleading where there is any real uncertainty associated with the task. A more realistic way of modeling tasks with uncertain scope is an open-ended distribution. Exhibit 2 shows such a distribution—the most likely duration is 10 days, there is a 90 percent chance that it will be completed in 15 days, and a 10 percent chance that it will take more than 15 days. There is no limit to how much longer that 10 percent chance represents, although the probability diminishes sharply for longer periods. There is a 1 percent possibility it could take longer than 20 days, a 0.05 percent possibility of longer than 25 days and so on.
Exhibit 1. This model explicitly denies any possibility of the task continuing beyond the upper limit.
Exhibit 2. This model allows for the possibility of exceeding the upper limit.
Using Open-Ended Distributions
In order to describe a task using an open-ended distribution, three quantities must be specified:
Base estimate—the cost or time required if everything goes according to plan
Contingency amount—the additional cost or time that may be required if things do not go according to plan, based on what can be reasonably foreseen at the present time
Overrun probability—the likelihood that the actual cost or time will exceed the allotted contingency.
The contingency amount allows for the possibility that the task may prove more complex or difficult than expected in the baseline project plan. To this extent, typical contingency levels of 10 percent to 50 percent are quite reasonable. However, if the scope of a task is not well established, it is difficult to set an appropriate contingency level because there is no way of knowing how much extra effort will be needed.
Overrun probability is intended to capture this information based on the degree of uncertainty surrounding each task (Exhibit 3). Establishing an overrun probability for a task does not set any hard limits to the extra cost or time that may be required for that task. However, because the probability that additional effort will be required falls off fairly rapidly, project managers can set reasonable overall cost or time limits for the project as a whole. Whereas contingency amounts apply to individual tasks, overrun probabilities enable project managers to set a project-level contingency.
In Exhibit 2, the base estimate is 10 days, while the contingency amount is five days, indicating a relatively large contingency, presumably to deal with possible complexities in the work to be performed. The overrun probability is 10 percent, indicating a medium to high level of uncertainty surrounding the task scope.
Overrun probabilities greater than 50 percent are not used because there is as much chance of exceeding the contingency limit as there is of staying within it—essentially this estimate is no better than a guess. If there is greater than a 50 percent chance, project managers should increase either the base estimate or the contingency amount to bring the overrun probability down.
In the case of a conventional, fairly certain task, the time to completion typically results from a combination of independent, noninteracting quantities. In this case, the time to completion is described by a normal distribution. For example, if a group of workers is digging a ditch, each one will work at a different rate depending on strength, motivation, and the density and compactness of each particular section of the ground. Because all these quantities are independent of each other, a histogram of the time taken for different groups to dig the same-sized ditch would approximate a normal distribution.
However, when the scope of the task is uncertain, the various factors affecting the time to completion are not independent. For example, uncertainty in the functional requirements for a software module will, in addition to the time taken to sort out the requirements, probably result in a longer time to develop a system design—because the design is likely to go through several iterations. Another way of looking at it is that, in such a case, there will be feedback between the requirements and the system design. This, in turn, leads to a longer time to code and test, and so on. Uncertainty in the requirements has a multiplier effect on the time to do everything else.
In these situations, the time to complete a task is made up of the product, rather than the sum of a number of random factors. This is exactly the situation described by a lognormal distribution. Whereas a normal distribution occurs when a series of random quantities are added together, a lognormal distribution occurs when a series of random quantities are multiplied together. Exhibit 2 is a log-normal curve.
Exhibit 4 compares Monte Carlo results using closed and open distributions. An engineering development project was used as an example. If each task had zero uncertainty associated with it—so that its duration would be exactly as predicted— the project would last 18 months. Of course, it would be unrealistic to expect this. The real question is: What would be a realistic completion date for this project?
While some project areas are fairly conventional with low uncertainty levels, a few tasks have medium uncertainty levels, and one or two have high uncertainty levels. Based on this information, Monte Carlo simulation can be used to derive a probable completion date—but the date will depend critically on how task uncertainties are modeled.
Closed distributions would assume that definitive limits could be applied to the duration of each task. Open-ended distributions assume probabilistic limits, such as a 90 percent probability that this task will be completed in a given number of days.
PSYCHOLOGICAL BARRIERS TO OVERRUN PROBABILITIES
In many organizations, the mere mention of the possibility of an overrun anywhere in the project is viewed as evidence of defeatism or lack of will—even “lack of moral fiber.” This attitude leads to unrealistic expectations, as if the mere setting of a delivery date makes it achievable.
The wise project manager will understand that in many projects, there are unknowns and imponderables, and these projects can only be brought to an acceptable conclusion by identifying such areas in advance and making allowances for them.
Setting any upper limit to an uncertain task is an act of faith.
Exhibit 4 shows the results of a Monte Carlo simulation of the completion date. The project start date was January 2000, and the project baseline completion date (without any allowance for uncertainty) was June 2001. Two different Monte Carlo simulations were performed, one using closed (triangular) distributions for each task, similar to Exhibit 1, and the other using open-ended (lognormal) distributions for each task, similar to Exhibit 2. The 90 percent probable completion date based on the closed distribution simulation is mid-December 2001, while the corresponding date for the open-ended simulation is late April 2002—a difference of 4.5 months, or almost 20 percent.
Unlike the closed distribution simulation, the open-ended simulation has considered the possibility that the uncertainty associated with some tasks may cause unforeseen schedule delays. This is, of course, an everyday project management reality and is the reason why the April 2002 date probably represents an achievable delivery date, whereas the December 2001 date is almost certainly impractical.
Realistic project plans must account for uncertainty. By using an open-ended model within a Monte Carlo simulation, uncertain tasks can be accommodated and a more realistic budget and completion date for the project as a whole can be developed. PM
Roger Graves, Ph.D., is an aerospace engineer, project manager and risk manager. He was a member of the team that rewrote the risk management chapter for the 2000 edition of A Guide to the Project Management Body of Knowledge (PMBOK® Guide). He is president of Davion Systems Ltd., an Ottawa, Ontario, Canada-based risk management services and software company.
Exhibit 3. This sample table shows chances of going beyond contingency factors based on the level of uncertainty, as defined from little to very high. Actual numerical values will vary from one industry sector to another.
Exhibit 4. Using open-ended distributions with Monte Carlo analysis of the project schedule gives a later and probably more realistic completion date than closed distributions. The schedule was set up using Microsoft Project, and Monte Carlo analysis was performed using Palisade Corp.'s @Risk for Project. Since the lognormal function in @Risk is not in a form which is immediately applicable to project management use, Davion Systems Ltd.'s RIAS was used as a front end to generate the lognormal formulas.
PM NETWORK | DECEMBER 2001 | www.pmi.org