Abstract
Projects play an increasingly important role in the business world today, and how an organization manages its projects remains critical to its success. Yet, the media routinely report stories of projects failing, in the sense that they are either delivered late, over-budget, or with reduced scope—a phenomenon known as the Planning Fallacy. We investigate project performance and the factors that contribute to the planning fallacy. Using data gathered from a large number of various types of projects, we develop a procedure for evaluating project outcomes, such as costs, completion times, and revenues, based on a combination of statistical and judgmental information that is widely available and updateable.
Keywords: planning fallacy; reference class forecasting; outside view; project estimation
Introduction
Projects play an increasingly important role in the business world today, and how an organization manages its projects remains a critical success factor. Yet, the media routinely report stories of projects failing or underperforming, in the sense that they are either delivered late, over-budget, or with reduced scope. Prominent recent examples are the development of Windows Vista, the construction of London's Wembley stadium, the development of the Airbus 380, and Boeing's 787 Dreamliner.
Although these cases are colorful, they are not abnormal. Our recent study of a large sample of public and private projects shows that, despite the increase in knowledge, expertise, and use of project management tools and techniques, many projects still underperform. This general tendency of projects to overpromise and under-deliver is called the Planning Fallacy (Kahneman & Tversky, 1979).
Daniel Kahneman and Dan Lovallo (Kahneman & Tversky, 1979; Kahneman & Lovallo, 1993; Lovallo & Kahneman, 2003; Flyvbjerg, Garbuio & Lovallo, 2009) suggest that the planning fallacy is the result of taking an inside view instead of an outside view. To take an inside view in project planning is to focus on the specific project at hand and estimate what will happen in that project. The outside view is focused on the project as a member of a class of projects and looks at how long projects of that class typically take.
A proposed method for overcoming the planning fallacy follows from Kahneman's initial insight that the key was to view a specific project as one of a class of projects. Put simply, the method Kahneman and Lovallo (2003) put forward is to develop a reference class (or sample) of projects similar to the one being evaluated, to establish a probability distribution for that reference class for the parameter being estimated, and then to compare the specific project with this reference class. Flyvbjerg (2008) and Flyvbjerg, Garbuio, and Lovallo (2009) propose and actually apply a simplification of this method. They looked at a large sample of projects and compared their predicted costs with their actual costs, and then provided an “optimism bias uplift” that would adjust costs for the optimism effect (Department for Transport, 2006). This approach, although it serves as one of the few attempts to actually implement the reference-based forecasting method, is limited in two ways. First, it is highly institution and domain specific. In fact, the uplifts they propose differ by country, domain, and project type. Knowing what the bias uplift is for single-span bridges, for example, will not help with estimates for double-span bridges. Second, the method produces perverse incentives—it is easy to imagine contractors underestimating costs by 20%, knowing that they will receive a 25% uplift.
In our study, we investigate project performance and the factors that contribute to the planning fallacy. Using data gathered from a large number of various projects, we relate time and cost overruns to a set of explanatory factors, with the objective of predicting the magnitude of possible overruns from the projects’ attributes and characteristics. Using such predictions, we develop a procedure for evaluating project outcomes based on a combination of statistical and judgmental information that is widely available and updateable. Our aim is first to identify most general explanatory factors, relevant to a wide variety of projects, to be used for providing a prescriptive method for decision-makers and planners to overcome the planning fallacy. Second, we develop a project classification scheme, which will enable parameter probability distributions to be generated for new projects.
Preliminary Data Analysis
Our preliminary analysis draws conclusions based on a 4,000-project dataset, from two sources. The data include information on a wide range of public and private projects, carried out in the United States and the United Kingdom between 1986 and 2008. The dataset includes information on initial project costs and duration estimates and actual costs and durations, in addition to project-based characteristics such as size, locations, teams, procurement systems, number of contractors, and so forth.
With regard to the first goal of our study, what causes time and cost overruns, we have made a number of observations. First, as perhaps can be expected, size is a reasonable indicator of whether or not a project will overrun. In addition, the experience of the management team makes a big difference. We also see, however, that our first dataset exhibits strong planning fallacy, in the sense that practically every project is over budget. Figure 1 plots the ratio between the forecasted to the actual cost, as a function of the forecasted cost. We see that for most of the projects, the cost ratio is below 1, and is the same at all project magnitudes.
Figure 1: Dataset 1—cost forecast/actual ratio versus forecasted cost.
We also find that cost estimates made in the later stages of a project are not necessarily more accurate and that the tendering contractor selection process and the procurement systems used during a project influence performance.
Interestingly, our second dataset exhibits little or no planning fallacy, either for costs or duration, as can be seen in Figure 2. Small time overruns are almost constant and invariant to project magnitude. These results call into question the universality of the planning fallacy and undermine the view that it is only due to cognitive bias.
Figure 2: Dataset 2—Duration forecast/actual versus forecasted duration and cost forecast/actual ratio versus forecasted cost.
Development of Outside-view Forecasting
Based on our findings so far, and through collaboration with several firms, we continue to develop our project classification scheme, which will enable parameter probability distributions to be generated for new projects, while making minimal use of subjective judgment and subjective estimation. Our approach combines cluster analysis, regression, and expert judgment and has the following advantages. First, it removes perverse incentives from the estimation process. Second, it changes the ranking of estimates, and therefore changes how resources are allocated. Third, the estimates derived following our approach are less likely to be subject to anchoring. We propose that our approach will add a fundamental element to the project forecasting toolbox, broaden the definition of reference class, and develop an objective method for project estimation.
