Project schedule risk analysis
Monte Carlo simulation or PERT?
by David T. Hulett
MANY PEOPLE HAVE USED a method of schedule risk analysis that was first introduced in the 1950s as the Program Evaluation and Review Technique (PERT). Some new versions of popular scheduling software have included a PERT tool intended for handling schedule risk. (Microsoft Project 98 and Scitor Project Scheduler 7 have PERT tools.) In a May 1999 PM Network article, titled “True Estimates Reduce Project Risk,” Mark Durrenberger presented a risk analysis of a simple schedule that is based on the Method of Moments (MOM). MOM was also the basis of PERT. (Note that PERT typically assumed that the uncertain durations could be described using a beta probability distribution, while Durrenberger and this article specify a distribution that is triangular. The choice of a particular distribution shape is arbitrary for PERT. The PERT approach, not the choice of a particular distribution shape, makes it inappropriate for the study of risk in complex schedules.) A Guide to the Project Management Body of Knowledge (PMBOK® Guide), in Chapter 11, Risk Management, calls the MOM “Statistical Sums.”
So, PERT, or its basis, the Method of Moments, is becoming more available to the general scheduler. The question is: “Does PERT adequately analyze schedule risk?” The answer, for all real schedules, is “No.”
The Durrenberger project has only one path, and PERT or MOM is a good approximation of risk in that limited context. In fact, project managers and planners should resist using PERT for risk analysis of any schedule that has more than one parallel path, which means almost all schedules of real projects, because PERT underestimates schedule risk in that more realistic context.
David T. Hulett, Ph.D., consults and trains on project risk analysis and management. He has presented papers on cost and schedule risk analysis and published papers on project schedule risk. He gives an annual risk management seminar for the Orange County PMI Chapter, the Primavera Users Conference, and the Performance Management Association. He is co-leading a task force to revise the PMBOK® Guide chapter on Risk Management. He is also an officer of PMI's Risk Management SIG.
Exhibit 1. Paths 1, 2, and 3 are exactly alike in logic, duration, and uncertainty ranges. (Paths 2 and 3 are collapsed.) They converge at the project completion milestone, causing the potential for the Merge Bias. The CPM duration of 17 days is shown for project completion.
Exhibit 2. Each converging path has this probability distribution. The average duration is 21 days, as Durrenberger predicted using the Method of Moments.
Why Conduct a Schedule Risk Analysis? A schedule risk analysis will help the project manager improve the project plan and identify and mitigate the risks to achieve a better outcome. (See “Schedule Risk Analysis Simplified,” Hulett, July 1996, PM Network.)
Many plans are (1) imposed on the project but are unrealistic from the start, or (2) derived from a good critical path method (CPM) schedule but using durations that are presumed to be accurate. Certainty is a scarce commodity in the uncertain world of most projects. A risk analysis looks at the schedule's durations with an honest skepticism, trying to identify in advance the uncertainties that may occur and what they may do to the schedule.
Starting from a good CPM schedule, a project risk analysis will help the project manager determine three things: (1) How likely is any date—but particularly the CPM date or any imposed date—to occur, given the current plan? (2) What is the exposure to risk, which is the same as asking, “How much schedule contingency do I need to drive the risk to an acceptable level?” (3) Which activities contribute the most to overrun risk in the project?
The risk analysis will give the project manager information to say to the customer, “My risk analysis indicates that the date you imposed on this project has only a 3 percent likelihood of occurring. You need an extra two months to bring this up to the 50–50 point. The main risks are in Unit 2, which is on the critical path 85 percent of the time.”
There are two main approaches to handling the schedule risk: Monte Carlo simulation and PERT. The Monte Carlo approach is a powerful tool, while PERT is severely limited in its application to one-path schedules.
What Is PERT? Like all schedule risk analysis, PERT assumes that the schedule logic represents how the project is going to be accomplished. Also, similar to schedule risk analysis, PERT focuses on the uncertainty of the activity durations.
The steps for using PERT are:
■ Use a three-point estimate to represent the uncertainty in durations. The three points are the durations under optimistic, most likely, and pessimistic scenarios.
■ Calculate the average duration and its standard deviation for each activity. The equation for the average depends on the distribution shape assumed. (The average for the triangular distribution is exactly (low + most likely + high) / 3. For the beta distribution, the average is typically approximated as (low + 4 * most likely + high) / 6.)
■ Compute the completion date for the total project by adding the average duration of the activities along the PERT critical path, which is the longest path through the network using those averages (see Durrenberger, May 1999, PM Network, for an appropriate example of this on a single-path network). The “PERT critical path” may be different from the traditional critical path.
■ Compute the standard deviation of the schedule path. (The standard deviation of the completion date is computed by taking the square root of the sum of the activity duration variances along the “PERT critical path.” Software packages such as Microsoft Project 98 and Scitor Project Scheduler 7 do not compute the standard deviation, so they provide, at best, an incomplete application of the standard PERT.)
■ Assume a shape for the probability distribution of the total project completion date and use it to determine the likelihood of any particular date happening, based on the average and standard deviations computed earlier. Durrenberger assumed the normal distribution, which is standard, but lognormal or other distributions have been assumed.
Exhibit 3. The risk of a project with parallel paths is greater at the merge point. Here, three parallel paths merge, creating an average duration of 24 days, compared to the CPM estimate of 17 days and the 21 days predicted by Durrenberger using MOM/PERT as shown in Exhibit 2.
The Perils of PERT. While PERT may represent each activity's mean duration and variance correctly, the entire schedule's risk is underestimated because PERT ignores the extra risk that occurs at path convergence points. Unfortunately, real schedules have many parallel paths and convergence points, so PERT is not an appropriate method for examining risk in real schedules.
Reader Service Number 054
Durrenberger cautions readers about applying the MOM to a “project with many parallel paths … due to path convergence. In those situations, a Monte Carlo analysis is recommended” [May 1999, PM Network, p.48].
He is correct in this warning, and the balance of this article illustrates the point that PERT is inappropriate for any real project.
The Merge Bias. Durrenberger gives a simple example of a one-path schedule of five activities. The traditional CPM path duration calculation is 17 working days. His PERT expected completion duration is 21 days.
The problem at points where schedule paths converge is that any of the several converging paths can delay the project. Hence, the project is more risky than even the riskiest of the incoming paths. This effect is so well known that it has a name: Merge Bias. A simple illustration will serve to make the point.
Suppose we have a schedule (shown in Exhibit 1) with three paths, each with the five activities that Durrenberger proposes, including their respective durations and ranges of uncertainty. Note that the 17-day duration is easy to track because we start the project on 1 September and allow the project (for example, a refinery turnaround) to work weekends.
A Monte Carlo simulation of this schedule shows the following results:
■ The result for a single path is shown in Exhibit 2. Since the paths are identical, we need to show the results for only one. It shows that the PERT approach is appropriate for the single-path schedule.
■ The result for the three-path schedule is shown in Exhibit 3. It shows that there is significantly more risk than in the single-path schedule and that the PERT approach underestimates risk, perhaps by a large margin.
The first results are for Path 1, although those for Paths 2 and 3 are identical. The simulation reproduces the Durrenberger result:
■ The CPM date of 17 September is 5 percent likely.
■ The PERT date of 21 September is 40–50 percent likely.
■ The average date is 21 September, as predicted by PERT.
■ A conservative date—choose, for instance, the 80th percentile—is 24 September, or one week later.
Exhibit 3 shows the risk of the total project, with the three paths converging at the project completion milestone. The risk of overrunning the 17 September date for the milestone is much greater than it is for any of the three paths. This result is the basis of Durrenberger's advice. (In these simulations, Microsoft Project 98 is used with a Monte Carlo simulation add-in: RISK+ from ProjectGear Inc. Other popular software that implements Monte Carlo simulation includes @Risk for Project from Palisade Corp. and Monte Carlo from Primavera Systems. Open Plan Professional from Welcom Technology also includes simulation capability.)
Reader Service Number 009
For the three-path merge-point project:
■ 17 September is not even on the radar screen, although at least one of the 5,000 iterations resulted in completion on 17 September. This project will almost certainly overrun unless the project manager changes the plan to mitigate the risk.
■ The average completion date is 24 September, significantly longer than the 21 September date predicted by the MOM/PERT.
■ In fact, 21 September that PERT computes, even for this three-path schedule is about 10 percent likely to occur.
■ The 80th percentile is now 26 September, two days later than it is for any of the three converging paths.
What's going on here? Why is the completion milestone less likely to be achieved on time than is either of its three predecessor paths? It is the working of the merge bias. For illustration of this effect, we will concentrate on the PERT/MOM solution—21 September—that is about 45 percent likely to be achieved for each path. The completion milestone is only about 10 percent likely to be completed as of 21 September. Why is this?
■ While Path 1 is 45 percent likely, only in 45 percent of those successful chances is Path 2 also successful. The likelihood of this occurring is (.45 * .45) = .20.
■ Only in 45 percent of the 20 percent likelihood of success along Paths 1 and 2 will Path 3 also be successful. The likelihood of this is (.45 * .45 * .45) = .09, or 9 percent likelihood of project success.
■ The more parallel paths that merge, the less likely is success by any date, in general (assuming that the two paths’ activity durations are independent and not correlated).
THE MERGE BIAS is the main reason that PERT and MOM fail to account sufficiently for schedule risk. This effect has been known since the early 1960s, shortly after PERT was developed. (In “An Analytical Study of the PERT Assumptions,” by K.R. MacCrimmon and C.A. Ryavec, Research Memorandum RM-3408-PR, The Rand Corp., Santa Monica, Calif., December 1962, the authors did not fully understand the mechanism at the merge point, but they raised the correct issue: that PERT ignored all but the longest merging path.)
For this reason, the PERT tool provided in some popular scheduling tools should not be used on real projects. PERT gives a false sense that a proper schedule risk analysis has been conducted. Perhaps worse, using PERT or MOM wastes the considerable effort expended in collecting the three-point estimates of activity duration ranges by providing a flawed tool.
Durrenberger is correct that the Method of Moments should not be used in real schedules that are likely to have many parallel paths and points of path convergence. A project schedule risk analysis done correctly using Monte Carlo simulation can provide the project manager with a great deal of information about the schedule. ■
Reader Service Number 057
PM Network February 2000