Decision analysis in projects

Sensitivity analysis is an important modeling tool. This provides ways to identify key variables that have the greatest impact on the outcome value and its uncertainty.

A non-probabilistic, or deterministic, cash flow model provides the basis for valuing each possible outcome, Monte Carlo simulation and decision tree analysis are ways to use probability distributions to represent judgments about risks and uncertainties as input variables. Either technique turns the determination model into a stochastic (i.e., probabilistic) model.

Correlation is a property of variables that are related. For example, an increase in one variable might be generally associated with an increase in another variable. This would be a positive correlation. In stochastic modeling, if any two variables are related, that relationship must be incorporated in the model; otherwise, the analysis will be wrong.

Dynamic modeling is a variant of most often, Monte Carlo simulation. In such models, subsequent decisions are made as time progresses. This represents, for example, a project manager who redirects and makes corrections along the way.

SENSITIVITY ANALYSIS

Project models are comprised of variables and formula relationships. Input variables are combined through the formulas to determine the predicted outcome and the corresponding outcome value. Obviously, some variables will be more important than others. Sensitivity analysis is the process of analyzing the relative importance of elements in the model. Usually attention is focused on the input variable assumptions.

Sensitivity analysis can be employed while developing either deterministic or stochastic models. With stochastic models, we have the additional influence of uncertainty. That is, variables having a high degree of uncertainty have a correspondingly greater influence upon project outcome uncertainty.

The purpose of sensitivity analysis is to identify key variables warranting special attention. The analyst may want to spend time in obtaining expert judgments in proportion to importance of the respective input assumptions. Typically, two to five input variables are the cause of 95 percent of the uncertainty in a problem analysis.

Single Variable Sensitivity

In the simplest application, sensitivity analysis is performed by making alternative trials of a deterministic model with step changes in one or more of the variables. These can be either chance or decision variables. Usually only one variable is changed at a time; the other variables are set at their base case value. This form of sensitivity analysis is easy to perform and does not involve probabilities.

A spider diagram, such as Figure 1, is a popular way to present sensitivity analysis. The x-axis is a percent or fraction deviation from the base case. Although any dependent parameter (PV IRR, etc.) can be used along the y-axis, an outcome value measure is preferred. The graph shows how sensitive the model's outcome is to changes in individual input variables: the more sensitive, the steeper the slope.

Figure 1. Spider Diagram

Figure 2. Tornado Diagram

Figure 3. Sensitivity Times Uncertainty

A drawback to the spider diagram is that the percent deviations say nothing about the range of uncertainty for each variable. A variant of this diagram extends the “legs” out to, for example, bounds of 80 percent confidence levels for the respective variables. Some practitioners further embellish the graph with “webs” connecting the data points,

A more popular way to express the effect of the range of uncertainty for input variables is the tornado diagram. An example is shown as Figure 2. Alternative cases are run, changing one variable at a time to a low confidence bound (e.g., 10 percent), then to a high confidence bound (e.g., 90 percent). The variables are prioritized in sequence of importance according to the range of the resulting outcome values. When the graph is oriented in the manner shown, the outline of the bars resembles a tornado. Note that the top four or five variables account for most of the outcome uncertainty.

Sensitivity Leverage

In the tornado diagram, Figure 2, two effects determine the importance of a variable to the analysis outcome value:

• The sensitivity of the model to changes in the variable's value (i.e., the slope of the line in a spider diagram), and also
• The uncertainty, or range, of the variable.

These two aspects have a leverage effect, similar to moment-arm multiplication, as shown in Figure 3.

Variable Interactions

Sometimes in performing sensitivity analysis, two or more variables are changed simultaneously to determine the joint, or combined, effect. The joint effect is usually different than the sum of the component variances. Analysis of joint effects is cumbersome with sensitivity analysis using the deterministic model.

There is a relatively new way to do sensitivity analysis that recognizes the possible interactions of variables and calculations through the model. A Monte Carlo simulation is run, saving input variable values for each outcome value. For each input variable, a correlation coefficient¹ is calculated. This represents the degree of association between the variable's random sample values and the project's trial outcome values.

The correlation coefficient ranges from + 1 if the variables have a perfectly positive correlation to -1 when they have a perfectly negative correlation. A + 1 would mean that the sequenced (ranked) input variable values and outcome values matched perfectly. If the correlation coefficient was close to + 1 or -1, then a formula could be used to express the dependent variable as a function of the independent variable. A zero correlation would mean there is no pattern (correlation) between an input and the outcome variable.

Figure 4 shows a sensitivity chart illustrating the relative importance of each input variable according to variable rank correlation^2. As with a tornado chart, input variables are prioritized by the width of the bar. A similar sensitivity chart can be generated by approximating each variable's contribution to the variance of the outcome variable.

Sometimes correlation is temporarily disabled for sensitivity analysis to get a better prioritization of input variables. In this example, Shipping Unit Cost is of minor importance but is highl_y correlated to Material Units (p= -.5 in the model). Thus, it might be disabled to understand and the sensitivity of material units.

MODELING CORRELATION

A well-known deficiency in deterministic project models is optimistic completion times. Standard PERT³ considers only activities on the deterministic critical path. Such simple models do not consider near-critical paths that have a probabilistic influence on project completion time. Further, simple approaches usually assume independence between activity duration times,

Figure 4. Sensitivity Chart

Figure 5. Scatter Diagram

Variables are said to be correlated when a change in one is associated with a change in another. Often, these relationships are the natural effect of common dependencies. For example, project labor costs are influenced by local wage rates. Some other common factors that would link their dependent variables include:

• Inflation (important mostly for longer-term projects)
• Availability of certain personnel and other resources
• Competence of the project manager and key personnel
• System or problem complexity, which may be unknown at the project start
• Estimation biases
• Changing project scope of requirements.

If data are available, the easiest way to determine whether data variables are related is to plot them on an x-y cross-plot, or scatter diagram. Figure 5 shows for a particular type project that Person-Hours to Complete Project is related to Project Team Size. The correlation coefficient is about +0.7. Usually our understanding of the system is sufficient to identify if a variable is dependent upon another.

Decision tree analysis often involves conditional probabilities. Typically, one or more chance variables are partiall_y dependent on another event outcome. This idea is always present in value of information problems presented in the fourth installment of this series. Information outcomes are correlated to an event of interest. That earlier article showed how probability trees and joint probability tables can represent correlation relation-ships between variables.

Several methods of representing partial correlation between variables are available to Monte Carlo simulation modelers:

• A dependent variable is a variable whose value is determined by some functions of one or more other variables, plus a random noise. The model is decomposed to a point where driver variables account for most of the dependent variables' behavior. This is the approach preferred by the author when the problem warrants the effort of greater model detail.
• A dependency coefficient, similar to a correlation coefficient and ranging from -1 to+ 1, can be used to weight the random variable sampling parameters. This technique requires that the variable be declared the “independent” variable (and it usually is), and the second variable is declared “dependent.” In sampling the dependent variable, the dependency coefficient determines the degree of influence of the independent variable's sampling parameter.
• Rank correlation coefficients can be specified between variable pairs. This is used in a matching process as the variables are sampled to reflect the degree of correlation.

DYNAMIC SIMULATION MODELS

In most Monte Carlo project simula-tions⁵ the uncertainties are resolved at the start of a simulation trial. Random samples of input probability distributions (trial values) are substituted into the deterministic model. The model is then solved to determine the predicted outcome and its value for that trial.

Many assumptions are built into an analysis. Decision points during a project can greatly complicate the model.

A dynamic model makes adjustments as time progresses. Decisions are based upon the then-current forecasts. Predictions about the future will be based upon both then-current conditions and apparent trends. Some example in-progress decisions include:

• Reevaluating whether to continue a project after test results
• Changing requirements to adapt to a changed business outlook
• Expediting activities to make up for project delays.

A decision tree analysis can explicitly represent only a few subsequent decision points. Solving such a decision tree defines choices that will be based upon event outcomes realized up to that point. A decision tree is perhaps the simplest form of a dynamic project model. However, several to many decisions will overwhelm a decision tree approach.

Much more detail and flexibility can be accommodated in a Monte Carlo simulation model. These dynamic simulation models are some of the most interesting to develop. Key decision points are placed where management interventions can occur. Such models reflect the adaptive nature of real-world project management. Projects can be modeled to be, in effect, “self-healing.” Example situations where dynamic simulation is applicable include:

• Shared resource loading
• Affect on activity duration and cost, dependent upon activity start versus original plan
• Effects of contingencies, which often ripple through the system (when things start to go wrong, often a chain of unpleasant surprises is set into motion).

Monte Carlo simulation is the only technique that can represent such diverse and important details in the decision model.

The key to dynamic modeling is representing conditional branching. Choices are made by decision rules in the project model, representing the company's decision policy or behavior. The result is more realistic project planning and evaluation.

CLOSING REMARKS

This article presents additional techniques for the project modeler's toolkit:

• Sensitivity analysis, which should be used in most analyses.
• Correlation between variables, which—where it exists-should be explicitly recognized in the model. Otherwise, the outcome value is often incorrect.
• Dynamic modeling, which recognizes subsequent decision points in the project. Decisions during a project can improve the project's value— both in the model and in reality.

Models and the predictions they generate are only as good as the input assumptions. The next article in this series describes biases and techniques for eliciting objective judgments about risks and uncertainties.

NOTES

1. The correlation coefficient is

The standard deviations in the denominator in effect normalize the covariance in the numerator. It is more convenient for Monte Carlo simulation modelers to work with the correlation of data pair rank orders, rather than with the actual data values.

2. Rank correlation uses the rank sequence of data values (ordinal scale data) rather than the values themselves (ratio scale data) in computing the correlation coefficient.

3. Program Evaluation and Review Technique. Activity completion times along the deterministic critical path are represented as probability distributions and are added together.

4. Decision Analysis in Projects: Value of Information. PMNETwork, October 1993, pp. 19–23.

5. Decision Analysis in Projects: Monte Carlo Simulation. PMNETwork, January 1994, pp. 30-36. ❑

John R. Schuyler, PE, CMA, is principal of Decision Precision®, an Aurora, Colorado, firm providing training and assistance in risk and economic decision analysis. Mr. Schuyler teaches Petroleum Risks and Decision Analysis worldwide in association with Oil & Gas Consultants international His services focus on modeling capital investments, acquisitions, and other corporate planning decisions. He received B.S. and M.S. degrees in engineering from Colorado School of Mines and an M.B.A. from the University of Colorado. His prior experience includes vice president and evaluation engineering with the nation's fifth largest bank, planning and evaluation analyst for a major oil company, and senior management consultant with a national CPA firm.

PMNETwork • October 1994