Putting quality in project risk management, part 1
Part 1: Understanding Variation
by Lawrence P. Leach
In this first part of a two-part article, let's look at two mistakes that can add confusion to your estimates and cloud decision-making.
W• Edwards Deming, the man credited with the postwar industrial revolution in Japan, stated, “It would be a mistake to export American management to a friendly country.” [Out of the Crisis, 1986, MIT Press ] His reasons for this harsh statement were grounded in the many mistakes management makes in attempting to control the system of production. Two of these mistakes may be institutionalized in current methods of project delivery, and must be dealt with before project managers can significantly improve project performance.
In The New Economics: For Industry, Government, Education [1993, MIT Press], Deming defines common-cause and special-cause variation: “A fault in the interpretation of observations, seen everywhere, is to suppose that every event (defect, mistake, accident) is attributable to someone (usually the one nearest at hand), or is related to some special event. The fact is that most troubles with services and production lie in the system. … We shall speak of faults of the system as common causes of trouble and faults from fleeting events as special causes.”
The differentiation is important to avoid two mistakes:
1. Ascribing a variation as “special cause,” when it in fact belongs to the system (“common cause”)
2. Ascribing a variation or mistake to the system (“common cause”), when in fact the cause was special.
The problem with making these mistakes is that they both increase project (system) variation.
Walter Shewhart, mentor to Deming, notes in Statistical Method from the Viewpoint of Quality Control [1986, Dover Publications, reprint of 1936 original] that the use of process control provides a “means of directing action toward the process” so that management can say, “If you do so and so, then such and such will happen.” Project managers must consider project delivery as a process and focus this predictive capability on the project schedule and cost.
Thinking About the Project Planning and Control Processes: Statistical Control. All project managers understand that there is uncertainty in estimating project task cost and duration. They also know that there is variation in the cost and duration of actual task performance. Despite these uncertainties, customers and management usually require project managers to predict the cost and duration of the project. Project managers must also make project control decisions during project execution to attempt to bring the project to completion within these time and cost estimates.
Lawrence P. Leach manages the Project Management Office for American National Insurance Company (ANICO) in Galveston, Texas, USA. Prior to this he was founder and principal of Advanced Projects Institute (API),which applies business tools to help clients improve their work processes and management systems. With 30-plus years experience as a project manager, he is a member of the Project Management Institute and the American Society of Quality Control.
Exhibit 1. Here's a probability profile of a typical project task (duration or cost).
Most people believe that the actual distributions for most project tasks look something like that shown in Exhibit 1. One of the subtle but interesting facts that comes to light when you think of project task estimates as probability distributions, such as Exhibit 1, is that any point estimate has a probability of zero. The idea of accurate point estimates does not have real meaning. In other words, any (and all) point estimates are equally “inaccurate.” For a discrete event—that is, the outcome of a specific project task (duration, cost)—all you can say is that, if this task were performed many times, some number of the trials would likely fall between two limits. Shewhart wrote that, “A statistician does not attempt to make any verifiable prediction about one single estimate; instead he states his prediction in terms of what is going to happen in a whole sequence of estimates.”
Shewhart and Deming point out that, from a scientific perspective, statistics provide the only basis for prediction. Deming stated, “It is only in the state of statistical control that statistical theory provides, with a high degree of belief, prediction of performance in the immediate future. … A statement devoid of rational prediction does not convey knowledge.”
Task estimating and many task performance processes are subject to statistical control. (They had better be; otherwise project plans are meaningless guesses!) Defining projects as “one of a kind” often leads to a false conclusion that project tasks are one of a kind. Most project tasks are done many times over by the people or contractors who do that particular work. Project estimating is a process performed many times. Specific project task output may differ (for example, the equipment ordered or particular line of code may differ, even though I am filling out the same order form or using the same software development process and language). But the basic task processes are most often repeated over and over by the performing agents.
Mistake 1. Treating Common-Cause Variation as Special-Cause Variation. Deming's funnel experiment, described below, demonstrates how the first mistake always leads to increasing the variation in the result. In projects, this is manifest by unnecessary changes to the project, leading to cost and schedule overruns.
The 1996 PMBOK® Guide chapters on project time and cost management (6 and 7), the section on project control (10.3), and the chapter on project risk management (11), do not identify the difference between common-cause and special-cause variation. While not identifying them as such, PMBOK® Guide Chapter 11, “Project Risk Management,” actually focuses on special causes of project variation, noting, “Potential risk events are discrete occurrences … .” However, it then goes on to describe methods for common-cause variation analysis: PERT (project evaluation and review technique) and project simulation. It does not differentiate between the two types of variation.
The PMBOK® Guide's more detailed discussion of project quality management, some of which is based on Ireland's Quality Management for Projects & Programs [1991, PMI] notes, “Many projects, because of their short-term and unique character, do not directly use statistical concepts … .” I fear that this statement may lead some to believe that it is not appropriate to do so. As noted above, this perception is flawed because (1) project planning and execution is itself a process performed many times by project organizations, and (2) project tasks are usually performed many times by the task performers.
The PMBOK® Guide's detailed discussion of project risk management, some of which is based on Wideman's Project and Program Risk Management: A Guide to Managing Project Risks and Opportunities [1992, PMI] also fails to differentiate between special-cause and common-cause variation. The experiment will demonstrate why discrete project risk analysis and mitigation methods should be separated from common-cause variation to improve project success.
Recent articles in PM Network by Hulett [“Project Schedule Risk Analysis: Monte Carlo Simulation or PERT?” February 2000] and by Ruskin [“Using Unders to Offset Overs,” February 2000] dealing with the problem of uncertainty continue the trend of failing to differentiate between common-cause and special-cause variation. Worse, they specifically discuss including special-cause (discrete, identifiable) risk events in revised distributions for Monte Carlo analysis. They encourage Mistake 2.
Exhibit 2 illustrates an experimental funnel setup, described by Deming, to cause an understanding of variation. (You can try this at home!) The experimental setup comprises a target on the floor, a funnel on a movable stand, and a marble to drop through the funnel. The aim of the funnel system is to cause the marble to land on the center of the target. For a project, you can think of the target as the committed project completion date (x) and cost (y).
The experimenters determine the capability of the funnel system (Rule 1) by centering the funnel over the target, and repeatedly dropping the marble through the funnel. They mark where the marble lands after each trial, resulting in the pattern illustrated by Exhibit 3 (produced by computer simulation). You might consider the x coordinate on the exhibit as the deviation in project task actual time compared to an estimate, and they coordinate as the deviation of task actual cost to estimated cost. (While cost and time often are correlated, in this simulation they are not. Cost and time may actually be negatively (accelerating project tasks increases cost) or positively (task time extension increases cost) correlated.)
Project managers carefully monitor task performance. When the critical path changes (because variation in task performance causes a project path parallel to become longer than the previously identified critical path), management usually shifts focus and expedites the new critical path. Or, if a task or the trend of tasks shows some variation (say, 5 percent), project managers frequently must report on the variance, including identifying the action they will take to recover the variance. Let's try this in the simulation by moving the funnel whenever the marble misses the target. We will move the funnel by the same amount, but in the opposite direction, from which our task cost and time missed the mark (which you may consider as the project baseline estimate).
Exhibit 4 illustrates results from the simulation with Rule 2. The spread of the points has increased! The increase was predictable because the variation was random to begin with. You are adding the funnel movement to the variation. The correction process increases the variance of the result. Since the funnel movements match the amount of movement of the marble, the variance increased by a factor of two. Variance is proportional to the square of the range of the distribution, increasing the spread of the points by the square root of 2, or about 40 percent. This is not a good management strategy.
Exhibit 2. The funnel experiment simulates common-cause variation. For a project, consider horizontal as schedule variation and vertical as cost variation.
Exhibit 3. Results from simulating the funnel when using Rule 1. Note that the funnel is not moved.
Rule 3 represents an alternative management strategy: Move the funnel back to zero each time before making the adjustment. In a project, this adjustment might correspond to updating the project baseline with a change, or adjusting the plan to actual before taking the management control action. You might think that people would never take action to “recover” from an underbudget or ahead-of-schedule situation. Think again. If someone on your project adds features to a project because they have the time or money, they are doing exactly that! If they put review of a project deliverable at a low priority because “it isn't due till the end of the month,” they are doing it. If they work to improve a software module that meets the specification because they “have time in the schedule,” they are taking action to “recover” from an underschedule situation!
Exhibit 5 illustrates the result of these actions. Note the expanded scale to accommodate the increased variation. The shape of this distribution varies with repeated runs of the simulation, but the general trend is to go off the chart in two directions. I believe that the explanation for projects that go seriously awry relates to this phenomenon.
Exhibit 4. Results from Rule 2, where funnel is moved from its present position.
Exhibit 5. Results from Rule 3: Move the funnel from the origin. Results from Rule 4 are similar, but sometimes run off the page in one direction at a faster rate.
Deming then tries one last strategy, which he calls Rule 4. Rule 4 places the funnel over the last position on which the marble landed. Deming compared this strategy to “worker training worker,” and to our precedent-based legal system. Shewhart noted, “When a judge makes a mistake, it becomes law.” In the world of projects, this might correspond to basing the estimates for your next project on the actual from the previous project. Results from this strategy are similar to Rule 3, trending over repeated application to wander off the paper unbounded. Compared to Rule 3, this rule usually runs off in one direction at a faster rate.
I hope these illustrations demonstrate that treating common-cause variation as if it were special-cause variation is a very serious mistake. It can cause great damage to project performance. Management manifests this mistake by management action, including expediting of tasks and processing project changes in an attempt to improve control. One should conclude that a means to separate common-cause variation from special-cause variation is needed before action is taken to “improve” project performance.
Mistake 2. Treating Special-Cause Variation as Common-Cause Variation. The second mistake leads to a failure to remove the special-cause of variation, leaving the potential for increased system variation. In projects, this is most frequently manifested as a failure to effectively manage discrete project risks. This failure means that risks are not effectively avoided or mitigated, again leading to project underperformance. Any risk you can identify (the first step of project risk management) is necessarily special-cause variation. You should handle such risks by prevention and mitigation.
Including special-cause variation in estimates for project task variation are also cases of Mistake 2. They cause you to overestimate project schedule contingency. When you include discrete risks in a PERT analysis or Monte Carlo simulation, you increase the predicted range of the project results; that is, you increase your cost and schedule estimate for a given confidence. This could cause you to lose bids for competitive projects. It could lead to actual increases in project time and cost through the natural tendency of work to fill the available time, and a use-it-or-lose-it philosophy on budget.
Recommendations. The reasoning described here leads to the following recommendations to reduce Mistakes 1 and 2:
Understand the difference between common- and special-cause variation.
Do not make adjustments to projects when the variation in project performance is within the range of common-cause variation. (If you do, you will degrade project performance.)
Do not include special-cause variation in your project risk simulations. (Understand project risk simulations as propagation of common-cause variation.)
Perform project risk management on discrete project risks (that is, potential special causes of variation) using discrete risk identification and mitigation processes, as described in the PMBOK® Guide and supporting literature.
These recommendations will reduce project variation and lead toward a competitive edge and greatly improved project success.
ONCE YOU HAVE a thorough understanding of variation, you should work to further reduce it by improving the processes for task estimation and performance. Part 2 of this article (coming in the March PM Network) will show you an effective method to do this. ■
PM Network February 2001