Introduction
Engineering infrastructure in Australia accounts for an increasing share of the construction market. In the 2005–2006 financial year, construction of infrastructure was valued at approximately A$40 billion, accounting for more than 12 percent of Australian GDP and employ 6.5 percent of the workforce (Engineers Australia, 2005). It has been estimated that, due to years of neglect, the immediate needs for infrastructure investment in Australia is about A$400 billion. The growth in infrastructure investment over the five years since 1999/2000 has been at 15.2% compounded annually (Engineers Australia, 2008). However, studies on infrastructure projects have found that large infrastructure projects are plagued by delays and large cost overruns. For example, studies have found that inaccuracy in cost estimation ranges from 20.4% to 44.7% depending on the type of projects (Flyvbjerg, Bruzelius, and Rothengatter, 2003; Flyvbjerg, Holm, and Buhl, 2002; 2005). Similarly, it has been reported that overruns of 50–100% in fixed prices are common for major infrastructure projects, and overruns above 100% are “not uncommon,” with the magnitude of cost overrun unchanged over the past 70 years (Bruzelius, Flyvbjerg, and Rothengatter, 2002).
Literature on the management of infrastructure projects point to 2 main culprits for the inaccuracy in cost forecasting for infrastructure projects--optimism bias and strategic misrepresentation (Bruzelius, Flyvbjerg, & Rothengatter, 2002; Flyvbjerg, 2006). Optimism bias occurs due to the tendency for people to be over-optimistic by overestimating benefits and underestimating costs (Lovallo & Kahneman, 2003), while strategic misrepresentation occurs when people deliberately misrepresent project costs and risks due to political, economic, and/or organisational pressures (Flyvbjerg, 2006).
The main consequence of both optimism bias and strategic misrepresentation in cost estimation is that, blinded by their belief that “results are determined purely by their own actions and those of their organisations” (Lavollo & Kahneman, 2003) and the ignorance or discounting of risks/uncertainties in the estimates, the project owners are unable to fully account for the uncertainty or risks in their estimates. Failure to fully account for risks will result in flawed planning, possible breakdown in relationship between the client and contractors, difficulties in delivering the project, and the inevitable project delays and cost overruns.
Flyvbjerg, Holm, and Buhl (2002; 2005) proposed using reference class forecasting (RCF) as a way to overcoming the effects of optimism bias and strategic misrepresentation in cost forecasting. RCF uses actual and estimated cost data from similar projects to determine the probability distribution of these types of projects. Based on the project estimators' preference for risk overrun, the cost estimates for new projects are then compared with the distribution curve from the RCF to determine the most likely outcome (Flyvbjerg, 2006). The main challenge for applying RCF method is the accumulation of a sample of similar projects with large enough sample size and accurate cost information. It may take a very long time for develop such a database. For some types of projects that are relatively rare in a country (e.g. nuclear power plants or large-scale desalination plants), it may never be possible to have a sample size large enough for statistical analysis. The problem is further exacerbated by the fact that private companies may not be willing to share such commercially sensitive information with competitors or the governmental agencies.
In the absence of such data sets for RCF forecasting, outsiders' view, a key contributing factor to the effectiveness of RCF, could still be brought in through a collective decision-making process that could potentially mitigate the effects of optimism bias and strategic misrepresentation. In this study, we examine the predictive validity of a cost estimation method that has been adopted by an increasing number of construction and consulting firms in their bids to improve estimating accuracy. The estimating method is called Risk Based Estimating (RBE). Despite its popularity, there is little empirical evidence on the predictive validity of RBE and there is a lack of understanding on how it improves estimating performance. The findings from this study identify the main performance drivers and suggest that the RBE method has good predictive validity.
This study is based on case studies of 11 water infrastructure projects in the Sydney region. The overall cost overrun/under-run data from the case projects provide a good indicator for the predictive validity of RBE. Further, through a series of interviews with selected people experienced in RBE, we have identified the main factors that drive improvement in RBE's predictive validity (hereafter referred to as performance drivers). In the next sections, relevant literature is reviewed and research questions defined, followed by a description of the research methodology and analysis process. Then, results are analyzed and conclusions are drawn. Finally, implications, validity threats, and further studies are discussed.
Literature Review
Infrastructure projects round the world are characterised by repeated and significant cost overruns (Bruzelius, Flyvbjerg, & Rothengatter, 2002; Niazi, Dai, Balabani, & Seneviratne, 2006; Raftery, 1994) across a full spectrum of projects, a phenomenon referred to as the “performance paradox” by Flyvbjerg, Bruzelius, and Rothengatter (2003).
Some academics have attempted to explain this paradox by referring to the uncertain and unique nature of construction projects. For example, McMillan (1992) states that “the uniqueness of the projects limits the learning process” such that wide project variability inhibits the practical benefits of past project experience. Others (Touran, 2006) believe that the multi-disciplinary nature of construction, coupled with the often long time periods between planning and completion, or even tender and completion, make predicting fluctuations in unit prices (Wen-der Yu, 2005) and accurately accounting for escalation (Touran, 2006), a difficult—and unreliable—process.
However, some authors have rejected any explanation that these cost overruns are a result of the difficulties associated with “predicting the future” Bruzelius, Flyvbjerg, and Rothengatter (2002). They believe that the “differences are too consistent and one-sided” for this to be the case, and instead state that it is the decision-making processes biased by political and management pressures that are the true source of the overruns. These authors argue that errors in estimates resulting from uncertainty are systematic biases that would improve over time as “errors and their sources…are recognized and addressed,” while psychological (such as optimism bias) and political biases are non-systematic and therefore likely to continue regardless of improvements in calculation techniques and data collection methods (Flyvbjerg, 2006; Fellows, 1996; Kujawski, Alvaro, & Edwards, 2004).
These non-systematic biases can be found on every level of the construction industry. For example, it has been recognized that in preparing estimates, estimators are likely to make “self-protective predictions,” influenced by their own perceptions of the competitive market regarding obtaining work and keeping clients (Fellows, 1996). Similarly, contractor tender prices are often a product of not only the estimating department but also managers' objectives; managers may reduce prices in an “ad-hoc manner” to unrealistic levels in an attempt to win the job (Kujawski, Alvaro, & Edwards, 2004). And finally, clients may strategically underestimate costs to ensure that the project goes ahead and to obtain funding (Mak, 2000).
However, this latter argument has 2 potentially flawed assumptions. Firstly, overcoming the systematic biases such as errors in estimating is contingent upon the industry having access to quality project data. Due to the competitive nature in the construction industry, contractors are unlikely to share their project data with others. Similarly, clients who may be exposed to political pressures, have their own interests in keeping project data to themselves. The data that is available is likely to be biased in that contractors and clients are more likely to release data from successful projects than unsuccessful projects. This means that the competitive and political nature of the construction industry make it very difficult for its members to develop a comprehensive dataset for use in improving their cost estimation accuracy by using methods such as RCF.
Secondly, Bruzelius, Flyvbjerg & Rothengatter (2002) also state that the paradox cannot be a result of the current cost estimating approaches. They reason that if the inaccuracies were merely due to the uncertain nature of the estimation methods, research into approaches specially developed to handle uncertainty would result in accuracy improvements. However, this view assumes that contractors and clients continuously improve their cost-estimating methods and are willing to try new techniques. However, no empirical evidence was presented to support the assumption and the opposite appears to be true: contractors continue to implement rudimentary cost estimating techniques that fail to adequately account for the uncertainties of the construction industry. For example, Baloi and Price (2001) comment that despite the close correlation between effective risk management and project success, risk is often accounted for by the addition of an arbitrary percentage contingency to their base estimate. In fact, it has been noted that “rarely do contractors quantify uncertainty and systematically assess the risks involved in a project” (Ali, 2005), instead preferring to rely on “assumptions, intuition and rules of thumb” (Mullholland, 1999). Wong and Hui (2006) noted that this problem is common among both contractors and owners.
In summary, there are various and sometimes contradicting explanations for the so called “performance paradox.” The effect of biases acts to distort the data used in the estimate, while the estimating method itself is often flawed due to failure to adequately account for the complex and often diverse risks associated with construction projects—suggesting that methods accounting for project risks have critical impacts on the predictive validity of estimates. In the following section, the focus is on Risk Based Estimating (RBE) and discusses how it could improve the predictive validity of estimating by addressing the 2 main arguments for continued poor estimating performance..
Risk-Based Estimating (RBE)
Range estimating (Shaheen, Fayek, & AbouRizk, 2007) is a simulation modelling process that is performed after a base estimate has already been made, to reflect the uncertainties associated with the estimate. One of the most common applications of range estimating is risk-based (Ali, 2005) cost-estimating, using Monte Carlo techniques. This method involves breaking the construction cost of a project into smaller components that are probabilistic (that is, non-deterministic) in nature. Each of these components is then quantified either subjectively (using the judgment of estimators, the project team, and risk experts) or objectively (using relevant statistics and data from previous projects) (Trueman, 2004), after which a distribution is selected. The final step involves running a Monte Carlo simulation—whereby the computer randomly samples very large numbers of potential combinations of risk and opportunity outcomes—that produces a probability distribution function (pdf) of the entire project cost by aggregating the component pdfs (Chau, 1995). The output pdf is most often a cumulative distribution probability graph that indicates the project tolerances—“the range of values above and below the estimated project cost…within which the final value is likely to fall” (Thompson, 1992)—and their associated probabilities, from which the project contingency is then selected.
The basic rationale behind RBE is that as a result of risks and their varying probabilities of occurrence, the projected final cost of a project is better depicted by a range of values (as a function of probability) rather than a single, deterministic figure. The size of this range, therefore, is a function of the amount of risk in the project. This risk is usually a combination of 2 types: (1) inherent risk, which is the uncertainty in the pricing or quantity of the known scope of work, and (2) contingent risks, which are risk events that may occur during the life of the project that differ from what has been assumed in the original pricing, such as safety and environmental risks (Trueman, 2004).
The RBE process usually follows on from the risk management process when the contingent risks are initially identified and evaluated (see Figure 1). Its primary purpose is to quantify the effect of both types of risk on the schedule (that is, project duration) and the budget (that is, project cost); only the latter will be discussed further, although the RBE process is similar for both and they are related. The basic steps of the RBE process, as shown in Figure 1, are discussed in the following sub-sections.
Figure 1. Risk treatment process
Base Estimate
The first step of the process involves preparing a base estimate, comprising the client's costs as well as the contractor's costs, which include indirect costs (margin, design, site establishment, insurance, and so forth) and direct costs (construction). These cost items are usually identified from information provided by the client (including drawings, contract documentation, specifications, and so forth), and often supplemented by assumptions made by the risk team—such as regarding certain elements of the scope—which are later tested during the risk workshop (see below). The cost of each of these items is then estimated using a combination of quotes and the expert estimator database.
Risk Workshop
The risk workshop involves the main stakeholders and any people with knowledge of the project and/or the potential risks involved, including the project manager, client, engineers, estimator(s), and any required experts, with the risk estimate manager facilitating. The risk workshop has a number of purposes, which include testing the assumptions made in the base estimate; identifying and evaluating any additional contingent risks that may have been missed in the risk management process (this also often includes identifying additional mitigation strategies); quantifying the potential effect of those contingent risks that could not be fully mitigated in the risk management process, known as residual risks; and quantifying the inherent risks.
The workshop participants use their personal judgment to quantify each of the inherent and residual contingent risks and opportunities, and must reach consensus, often with the aid of the workshop facilitator. Quantification of inherent risks and opportunities usually involves identifying the likely range of costs for each item of the base estimate (sometimes as a percentage of the base estimate but more often as a dollar value) using 3 parameters: the most favourable, the most likely, and the least favourable outcomes. Similarly, each of the contingent risks is quantified not only in terms of consequence, but also likelihood of occurrence (discussed in the next section).
The Risk Model
The risk estimates obtained in the risk workshop are then added to the risk model, an example of which is shown in Figure 2. The risk model is constructed in Microsoft Excel, and uses @risk to perform the Monte Carlo simulation. It is usually comprised of several spreadsheets, including at least an Inherent Risk sheet and a Contingent Risk sheet. Each of these contains a list of “line items”—a cost item on the inherent risk spreadsheet or a risk event on the contingent risk spreadsheet.
Figure 2 is an excerpt from an inherent risk spreadsheet. Each cost item is described using a quantity and a rate, the multiplication of which gives a deterministic figure, the net amount. The potential uncertainty associated with these quantities and rates is quantified using a range of percentages that are multiplied with the nett amount for each line item to give the range of cost. This cost range is assigned a probability distribution, the selection of which is based on the nature of the cost in consideration. Often a Pert or Pertalt distribution is used; a Pertalt distribution has been used in Figure 2.
The advantage of a Pert and Pertalt (and triangular) distribution is that they can be easily quantified using the three parametres mentioned above. For example, the P10, P50, and P90 values in Figure 2 represent the best, most likely, and worst cases for a Pertalt, while if a Pert distribution was used, P0, P50, and P100 are used. The use of P10 rather than a P0—and a P90 rather than a P100—in the Pertalt distribution accounts for the fact that the workshop participants are unlikely to have experienced the very worse and very best cases. This means that the model samples 10% above the worst case, the P90, and 10% below the best case, the P10, during the simulation, producing a more conservative result than when Pert distributions are used.
Figure 2. Excerpt from an Inherent Risk Spreadsheet
The contingent risks are quantified differently from the inherent risks because of the inclusion of an extra factor, the likelihood of occurrence. The contingent risk spreadsheet uses a binomial expression and an “if statement” to quantify the risks. The binomial is expressed as the function Riskbinomial (n, p), which specifies a binomial distribution with n number of trials and p probability of success on each trial. In Figure 3, an excerpt from a Contingent Risk spreadsheet, the yellow-outlined cell contains “Riskbinomial(1, D38),” where D38 is a cell referencing the likelihood of the risk eventuating; 50% in this case (green-outlined cell). If in a certain iteration, a percentage greater than 50% is returned, then a value of 1 is displayed (that is, the event occurs) in the yellow box, while if a percentage of less than 50% is returned, then 0 is displayed (the event does not occur). The blue-outlined cell next to the yellow-outlined cell contains an if statement, “IF(L38=1,RiskPert(I38, J38, K38),0)”; this means that the blue-outlined cell only returns a value when the yellow-outlined cell returns a 1. In this case, the returned value always lies somewhere on a Pert distribution described by the 3 parametres: the minimum, most likely, and maximum values, shown in the black-outlined cells. In summary, there are 2 possible outcomes for a contingent risk item: no cost if the risk event does not occur, or some cost if the risk event does occur, the cost lying somewhere between the minimum, A$0, and maximum values, A$200,000.
Figure 3. Excerpt from a Contingent Risk Spreadsheet
The Estimate
Once the risk model has been built, a Monte Carlo simulation is run until convergence in the results is obtained; usually 5000 iterations are used in order to obtain a smooth distribution curve. The cumulative distribution probability graph, also known as an “S Curve,” allows direct selection of a budgetary figure based on the probability of occurrence. As an example, Figure 4 shows that there is a probability of 90% that the total out-turn cost will be less than $52.4 million, and therefore, the client will have a risk exposure of only 10%.
Figure 4. A Cumulative Distribution Probability Graph
Methodology
Because of the lack of understanding on the predictive validity of RBE and the key factors affecting it, a case study approach was adopted. A case study approach allows flexibility in exploring the relevant factors that are contextually dependent and narrowing down to the in-depth understanding of key factors (Yin, 2002; Goubil-Gambrell, 1992). Both quantitative and qualitative data are collected from multiple sources, including project reports, interviews, and observations. The data was then triangulated whenever possible.
Case Selection
The case studies were chosen from a pool of water infrastructure projects owned by the Sydney Water Corporation (SWC) and the Sydney Catchment Authority (SCA), with all estimates performed by a single estimating team, and within the past four years (2003–2007). These cases were chosen for 3 reasons. First, publicly-owned organizations are more likely than their private counterparts to provide project data. Second, the use of 1 estimating team ensures that consistency in the implementation of RBE, minimizing the possibility that differences in implementation could affect the findings. Finally, as all the estimates had been performed since 2003, it was likely that all relevant data could be obtained and involved parties contacted.
In total, 11 projects were selected. To control for the effect of scope changes on the predictive validity of RBE, projects with numerous and significant scope changes were eliminated. Similarly, the projects selected reflect the range of project types (including sewerage treatment plant construction, augmentation, upgrades and decommissions, pumping station construction and upgrades, and storage transfers), sizes ($4 million–$120 million) and common procurement methods (D&C and Alliance) undertaken in the water sector in Australia. Therefore, “maximum variation sampling” guided our selection of cases (Goubil-Gambrell, 1992), allowing identification of common patterns emerging from a diverse group of projects.
Data collection
The majority of the quantitative data was obtained from the project owners or managers and the estimating team, using a combination of email messages, phone calls, and informal meetings with the project/alliance managers. The primary goal of interviews is to elicit interviewees' opinions and experiences in their own terms (Maxwell, 1994). Each interview took between 40 and 70 minutes, with the presence of 2 investigators. Interview questions were based on a pre-developed semi-structured interview protocol, as well as the responses from the interviewees. In addition, respondents were also encouraged to raise matters they consider relevant to the research questions.
In order to present the entire perspective of the construction industry and to ensure data triangulation (Goubil-Gambrell, 1992), interviewees included contractors, clients, and consultants. In total, eight people were interviewed; representing 2 major Australian contractors, 3 consultancy firms, and 2 client organizations. The selection of interviewees was based on the criteria that they have extensive experience (at least several years) in the use of RBE, as well general project management and costing experience of at least 10 years. The latter requirement was important for the interviewees to be able to adequately identify the key drivers of the RBE process, and to contrast it with traditional estimating methods.
Analysis
The analysis processes for the qualitative and quantitative data are elaborated below.
Qualitative Data
The relevant points or quotes from interview transcripts were identified, extracted, and categorized into several key categories, aided by data reduction and display. Conclusions were drawn by a process involving the identification of key themes, patterns, irregularities, relationships, and explanations within the data (Miles & Huberman, 1994).
Quantitative Data
Between the time that an estimate is made and the final project is delivered, a wide spectrum of project factors may cause the initial estimate or actual cost to deviate significantly from each other. Since the concern here is only the predictive validity of RBE, it is important that factors not related to RBE, such as large variations due to significant scope changes, are accounted for. As a result, the costs and risks included in the model and the final cost often needed to be adjusted. For example, the final project cost was often provided excluding client costs. This meant that client costs had to be removed from the model, and the model run again to produce a new cumulative probability distribution graph—referred to as the adjusted risk curve. Similarly, those variations that are the result of risks not included in the model are subtracted from the final cost, producing the adjusted final cost. These subtracted variations were usually the result of risks that were very unlikely and unforeseeable, and were therefore not included in the model, because to do so would result in an exaggerated estimate.
Measurement of Predictive Validity
The predictive validity of RBE was measured in this study in 2 ways. First, it was measured as the percentage difference between the final contract amount (that is, construction costs plus variations) and the contract award amount (that is, base estimate plus contingency), also known as “cost growth” (Baloi, 2001).
Second, because the selection of the budgetary figure was based on commercial considerations regarding how much risk the client or contractor is willing to take, and is unrelated to the accuracy of the estimate, it was sensible to compare the adjusted final cost to the cumulative probability curve, rather than the deterministic budgetary figure. The curve, however, represents the entire possible range of costs for the project. Therefore, as long as the final cost falls within the curve, the estimate was considered to have good predictive validity (see Figure 5). In practice, it was difficult to identify values at P0 and P100. Consistent with the PertAlt distribution function, we use P10 and P90 of the adjusted budget cure to gauge the predictive validity of RBE.
Figure 5. Relationship between the Adjusted Final Cost and Estimate Curve
Adjusting for Contract Types
Since risks are treated differently in D&C and Alliance types, the measure of predictive validity needed to be adjusted to each project type. Below, the reasons for and the processes of adjustment are discussed.
In an alliance, the TCE (also known as the “contract value” for the alliance) accounts for all the risks of the alliance. In other words, the TCE includes both the client and contractor risks. This means that all the parties must work together to keep the final cost within the TCE, as all their risks contribute to the final cost. The client can make additional requests, such as increases to the project scope, and the contractor will attempt to meet them so that they do not add to the final cost of the project. This is because the benefit of coming in under the TCE and within time is likely to be much greater than any benefit that could be obtained by the contractor maximising any client changes. As a result, there tend to be very few variations in alliances.
The TCE value is usually chosen to be equal to the P50 value from the TCE curve; this low probability figure is usually chosen because it is reasoned that the working relationship between the alliance partners will usually act to keep the cost down. A budget figure is also set, and usually includes those risks in the TCE plus program costs. As both the TCE and budget include the risks of everyone in the alliance team, and because there are usually no variations, the performance of the alliance estimates can be determined by comparison to either the budget or the TCE curves.
In a D&C, the contract value is merely the value for which the contractor agrees to perform the job (that is, only design and construction risks), and is usually determined in a competitive bid environment. It is not a value obtained using RBE and may, in fact, not lie at all on a cost probability curve of contractor risks. The budget figure for a D&C, though, is obtained in a similar way to an alliance in that it is calculated including all the risks (design, construction, project management, program, and client costs) and is usually selected to be the P80 or P90 value from the curve.
In contrast to an Alliance, there is a clear division of risk between the parties of a D&C: the contractor only bears the risk for the design and construction of works outlined in the contract. This, in conjunction with the adversarial nature of D&Cs, means that any client-initiated changes will automatically become variations, contributing to the final cost of the project—the contractor has no contractual obligation to minimise the effect of the variations to the client. So, the final contract cost includes the contract value plus any variations. As these variations are inevitable, the predictive validity of the estimate can only be determined by comparing the budget curve with the adjusted final cost. If the contract value was the yard stick, D&C projects would always experience overruns because of the variations.
Therefore, the performance of the alliances can be determined using either the TCE or budget curves, while the D&Cs must be evaluated using the budget curve.
Results and Discussions
The results are presented in four steps. First, the findings on the predictive validity of RBE are reported. Second, analysis of the main performance drivers of using RBE is presented and the findings are highlighted. Third, the implications of adopting the RBE method are discussed. Finally, validity threats are discussed and further studies suggested.
Predictive Validity of RBE Forecasting
The results reported in Table 1 show that, measured by the ratio of adjusted final cost to TCE, out of the six alliance projects, four are within or on budget and 2 overshot budget, 1 by 0.4% and the other by 4%. For the five D&C projects reported in Table 2, all are within budget, 2 by margins of more than 10%. Assuming the 0.4% overrun in project C is within the margin of error of estimation, then the results show strong evidence for the predictive validity of RBE method.
Table 1. Summary of Alliance Project Results
| Alliance Project | Target Cost Estimate | Adjusted Final Cost | Adjusted Probabilities | % of Final to Estimate | |
| P10 | P90 | ||||
| A | $117,953,998 | $117,876,092 | 115.0 | 120.5 | 99.9% |
| B | $45,500,000 | $45,505,000 | 45.0 | 46.1 | 100.0% |
| C | $54,594,319 | $54,807,270 | 53.4 | 55.7 | 100.4% |
| D | $4,714,538 | $4,560,944 | 4.54 | 4.9 | 96.7% |
| E | $14,609,906 | $14,724,029 | 14.19 | 15.07 | 100.8% |
| F | $37,179,400 | $38,661,830 | N/A | N/A | 104.0% |
Table 2. Summary of D&C Project Results
| D&C Project | Budget Estimate | Adjusted Final Cost | Adjusted Probabilities | Adjusted Probabilities | |
| P10 | P90 | ||||
| G | $32,200,000 | $31,328,000 | 28.6 | 32.2 | 97.3% |
| H | $58,900,000 | $56,000,000 | 49.58 | 58.9 | 95.1% |
| I | $32,600,000 | $27,000,000 | 29.2 | 32.6 | 82.8% |
| J | $51,000,000 | $48,430,000 | 43.25 | 50.3 | 95.0% |
| K | $65,000,000 | $58,250,695 | 51.0 | 65 | 89.6% |
The results reported in Table 1 and Table 2 also show that of the 11 projects studied, 10 have a final cost that lies between P10 and P90 on its adjusted budget curve, indicating good predictive validity of the RBE method. Please note that the adjusted P10 and P90 values for Project F are not included because of difficulty with obtaining the working risk model from the case organisation.
Key Performance Drivers of RBE
Through interviews and observations of the RBE process, 3 key performance drivers were identified as that contributing to the effectiveness of RBE as an effective cost estimation method. Each of these drivers are discussed in the following sections, focusing specifically on the way in which they mitigate the 2 main causes of cost overruns—bias and the inadequate account of project risks—identified earlier.
Collective Experience and Outside View
Lavollo and Kahneman (2003) suggested that in order to improve the reliability of forecasts, companies should take an “outside view” of the project at hand, by comparing it to a reference class of past similar projects. They argue that this outside view approach “counteracts the personal and organizational sources of optimism” that act to bias the forecast.
One of the ways in which this “outside view” can be achieved is when the decision-makers “detour into pertinent outside-view information” (Lovallo & Kahneman, 2003), as is done in the risk workshops. By involving “a range of different people with different skills, who generally know more about the project than us [the estimating team] coming in from the outside,” the workshop permits the development of “a much better understanding of the real risks associated with the project.” Compared to conventional cost estimation, where project stakeholders withheld information from others, the RBE process facilitates the sharing of information among project stakeholders.
During the workshop, information is elicited from the participants and incorporated into the estimate by “breaking the project into its constituent parts.” The participants then relate each of these project components to other projects in which they have experience, and in which that particular component (that is, risk) was an issue. As an example, Project J was a “one-of-a-kind” job and, as a result, “you couldn't just think about the whole project because it was new and hadn't been done before … But 90% of a project will actually be similar to other projects, if you think of it in terms of its individual parts.” This highlights the effectiveness of RBE in forecasting the cost of innovative projects, echoing Lavollo & Kahneman's opinion that “the outside view's advantage is most pronounced for initiatives that companies have never attempted before” (Lavollo & Kahneman, 2003).
In summary, the use of collective experience is believed to mitigate the effects of both optimism bias and strategic misrepresentation on the estimate in 2 ways. The first relates to the concept of aggregating multiple opinions in order to prevent any one person's opinion dominating the estimate; the rationale is that “the more stakeholders you bring together, the more information you bring together … the less personal opinion is reflected.” The second relates to the way in which it allows the workshop participants to contextualise the project at hand in light of their past experience.
Attention Focusing
According to Lyytinen, Mathiassen & Ropponen (1998), the use of an appropriate “framework can be used to direct managerial attention, to observe largely ignored risk areas and to orchestrate action.” This concept forms the basis of the second key performance driver, “attention focusing,” which refers to the way in which RBE focuses the attention of the project stakeholders on the critical risks of a project. The following discussion addresses some of the key ways in which RBE achieves this focus, using examples both from the interviews and from observations of the RBE process.
One of the most important and most obvious ways in which RBE focuses attention is through the systematic identification and quantification of project risks during the workshop. This process highlights the most significant risks, by reference to a combination of their consequence and likelihood of occurrence. As explained by one interviewee, this process “basically gives you a score” by which to rank the risks according to whether they are “significant, high, moderate, or low.” This ranking then focuses the remainder of the workshop(s), ensuring that the participants account for the most important risks and opportunities first. The next step involves sorting the risks “in order of significance … we start with all the significants, then we go to the highs, then the moderates and then the lows if we have time.” This ranking and sorting process was found to be a common practice across all groups of the industry.
Similarly, RBE also focuses the attention of the management team on the project scope. The workshop process “requires us to have a thorough knowledge of what we are doing”; in other words, “you have got to know your scope.” Lack of a detailed scope means that the costs and risks of the project cannot be specifically or confidently identified and quantified. To compensate, large risk ranges are required to take into account the extra uncertainty. In turn, this produces a wide cumulative probability curve, and potentially the selection of an unnecessarily large budgetary figure. The workshop highlights these areas of uncertainty in the scope, which can then be clarified, before completing the estimate. “What we are really trying to do [in the workshop] is to flush out uncertainty,” such as to “weed out scope which is not defined correctly.” This produces a “tighter” s-curve and also reduces the potential for future cost blowouts as a result of significant variations during delivery. As succinctly summarised by one interviewee, “the more you plan the job, the better it goes.”
RBE may also encourage the management team to focus on the opportunities—not just the negative risks—associated with the project. Lyytinen, Mathiassen & Ropponen (1998) remarked that there is a common behavioural characteristic among management to focus on the negative, rather than the positive. Specifically, they stated that in the minds of managers, “uncertainty of positive outcomes is not behaviourally an important aspect of risk.” Instead, “a risky choice is one that contains a threat of a poor performance.” However, because RBE costs both the negative risks and the positive uncertainties—the opportunities—into the budget, the project team must not only attempt to mitigate the negative risks, but also strive for the opportunities. For example, as stated by one interviewee, “sometimes we will have … million-dollar opportunities which will reduce our target cost.” While previously, although such an opportunity may have been recognised, it was often just regarded as “a good idea back then” and rarely pursued. However, as the RBE process builds these opportunities into the model and thus the budget, the management team must really “go after them or … go over budget.”
The majority of interviewees see RBE as “really a focusing tool.” The consideration of the specific risks and opportunities that are used in the model is a key factor that sets RBE apart from the traditional contingency method, which relies almost purely on the “gut feel” of the managers in assessing the risks. This difference in the focus on risks was cited by many interviewees to have a significant effect on the accuracy and reliability of the estimate. A former executive at a major U.S. construction company, which has had extensive experience in using RBE method, noted that “sometimes we got it right and sometimes we got it wrong, but since we have been using this (RBE) method here … we have been doing really well with meeting our cost targets and schedules.”
Implementation of Intuitive Modelling Techniques
The third performance driver of RBE relates to the modelling techniques and statistical concepts used to produce the final estimate, in particular, the use of 3-point estimates.
The interviews revealed that the use of 3-point distribution techniques is common—if not universal—across the industry. The main appeal of the 3-point approach is the ease with which workshop participants can transform their knowledge and experience into a pdf for input into the model: “It's useful because the things that you're trying to estimate, you often can't get representative data … so a 3-point estimate is a simple way of developing the distributions.”
It is not only “intuitive,” but “there's a lot of flexibility with the 3-point estimate.” With the same data, the estimating team can apply a variety of different distributions, such as triangular distribution and Pert distribution. Pertalt distribution appears to be a popular choice because “it's unlikely that each participant has experienced the very worst” and very best outcomes for certain risks, particularly those that are less common, while Pertalt accounts for this by sampling 10% above the worst case and 10% below the best-case estimate given by the participants.
This flexibility of 3-point estimates is also demonstrated by the variability in distributions used across the industry. The contractors prefer the triangular distributions, while the consultants prefer to use Pert and Pertalt distributions. Despite this difference, none of the interviewees believed that the type of 3-point distribution used had any discernible effect on the model outputs. However, distribution selection is a source of contention in academic circles. Some academics believe that several of the 3-point distributions are “quite accurate” at estimating construction costs (Keefer, 1994), while others believe that the use of certain distributions, in particular triangular, can introduce significant bias into the estimate (Adams, 2006; Chau, 1995).
The S-curve generated from the modelling process shows the entire range of possible project costs against their probability of occurrence and provides a comprehensive overview of the entire project—including the worst, most likely, and best outcomes. In contrast, a single deterministic figure—the output of the majority of estimating processes—does not provide “an understanding of the rest of the project; you don't know what your target or … best case should be. You just have one number and you don't really know where that sits on a range of possible outcomes.” Importantly, the S-curve enables the management and decision-makers to select from a range of budgetary figures contingent upon their preference of risk exposure levels (cost overrun).
For clients, RBE gave them “greater confidence in the estimate” and ensures “certainty of funding.” For example, previously, when requesting funding, if the budget requested was considered too high, the reaction would be “’You don't need that contingency, what's that for? … How do you justify this amount?' and they'd take it out so they could get a number they liked … Now, the project was doomed for disaster.” Under RBE, the budget amount corresponds to its probability of exceedance and the items included in the estimation model—showing that the value is not “just a buffering amount, but calculated risk.” The fact that RBE is based on a rigorous and transparent process and that the consequence of any budget reduction can be easily quantified from the S-Curve, makes any arbitrary budget reduction difficult to justify.
In summary, the intuitive and flexible nature of the statistical techniques used in RBE contributes to the performance of estimates in 2 ways. First, it allows participants to easily and reliably transform their experience into quantitative data that can be directly input into the model. Second, the ease with which the outputs can be interpreted by senior management, funding bodies, and insurance companies ensures that project owners and managers are able to obtain the correct amount of funding in the first place.
Conclusions
This study examines the predictive validity and the performance drivers of using RBE in water infrastructure projects. Results from 11 water projects show that projects using RBE have good predictive validity. Of the 11 projects, 9 are under budget and the remaining 2 are over budget by less than 1%, well within the margin of error.
Through interviews and observations, the following 3 main performance drivers for the RBE method were identified. First, the operation of RBE (especially during the workshop) requires the active participation of all major stakeholders to fully assess cost items and potential risks. The collective experience and “outside view” brought in by various stakeholders help to mitigate estimating bias and motivations for strategic misrepresentation.
Second, the quantification of risks and the consequent dollar value attached to a particular level of risk exposure help “shape” project stakeholders' attention to high priority/impact risks. We borrow from Lyytinen, Mathiassen & Ropponen (1998) and call this benefit “Attention Shaping.” Attention shaping ensures the focus on key risks and ultimately the delivery of projects within budget.
Finally, the data modelling techniques used in RBE are intuitive and simple to use. Central to the techniques is the 3-point modelling techniques that allowed the participants to easily transform their expert opinions into input data that are used to model the probability distribution of project costs.
