The accuracy of hybrid estimating approaches? Case study of an Australian state road & traffic authority
Reference Class Forecasting (RCF) is increasingly being adopted to mitigate two main factors causing persistent cost overruns in infrastructure projects—optimism bias and strategic misrepresentation. In practice, organizations tend to tailor generic estimating approaches to their unique operating environments by developing hybrid approaches blending RCF with other approaches such as the conventional fixed contingency approach. Yet, little empirical evidence exists on the accuracy of RCF or its variations. This study presents the results of estimation accuracy of a sample of road projects conducted by an Australian State Road & Traffic Authority using a hybrid estimating approach blending primarily RCF with fixed contingency approach. The results are then compared with historical results from literature on infrastructure projects and samples of two other dominant estimation methods, namely the conventional fixed contingency approach and risk-based estimating (RBE).
This study finds that the average accuracy of this sample using the hybrid approach compares favorably to historical results, which typically use the fixed contingency approach. This finding is further supported through comparison with a sample of conventional construction projects using a fixed contingency approach that shows the hybrid sample tends to produce estimates that are more likely to lead to under-budget rather than over-budget with similar dispersion around the mean estimation error. Additionally, the hybrid sample is compared with a sample of water projects using the RBE method to explore the differences in accuracy between the two methods. The results suggest that RBE is more accurate than the hybrid approach. The findings have important implications for both practice and future research.
The cost overrun of large infrastructure projects has remained unchanged during the past 70 years (Bruzelius et al., 2002). Studies report that inaccuracy in cost estimation ranges from 20.4% to 44.7% depending on the type of project. Overruns of 50% to 100% in fixed prices are common for major infrastructure projects, and overruns above 100% are not uncommon (Flyvbjerg et al., 2003; Flyvbjerg et al., 2002; Flyvbjerg et al., 2005a). Two of the main factors leading to inaccurate cost forecast for infrastructure projects are optimism bias and strategic misrepresentation (Bruzelius et al., 2002; Flyvbjerg et al., 2006). Optimism bias relates to the tendency for people to be over-optimistic by overestimating benefits and underestimating costs (Lovallo and Kahneman, 2003), while strategic misrepresentation refers to deliberately misrepresented project costs and risks for political, economic, and/or other gains (Flyvbjerg et al., 2006).
The main consequence of both optimism bias and strategic misrepresentation in cost estimation is the failure to fully account for risks/uncertainties in the estimates, due mainly to the belief that “results are determined purely by their own actions and those of their organizations” (Lovallo and Kahneman, 2003). As a consequence, flawed planning, breakdown in the relationship between the client and contractors, and difficulties in delivering the project, result in both project delays and cost overruns.
Kahneman and Lovallo (1993) conclude that the only way to mitigate optimism bias and strategic misrepresentation is to take the “outside view” when making decisions. The “outside view” typically comes from objective past performance data or the opinions from stakeholders who do not have a vested interest in getting the project up and running. Reference Class Forecasting (RCF) was introduced recently as a technique that mitigates optimism bias and strategic misrepresentation. It utilizes a database of actual performance of comparable past projects within a given reference class to provide an objective reference point for the cost forecast of a current project (Flyvbjerg et al., 2005a). Based on the distribution of overrun and project estimators’ preference for risk of cost overrun, percentage uplift is determined and added to the base estimate as risk contingency (Flyvbjerg et al., 2002). The main challenge for applying the RCF method is the accumulation of a sample of similar projects with large enough sample size and accurate cost information (Napier and Liu, 2008).
Despite the increased adoption of the RCF approach, little empirical evidence exists on the accuracy of RCF and how it compares to other dominant cost estimation methods. In practice, organizations tend to tailor generic estimating approaches to their unique operating environments by developing hybrid approaches, blending RCF with other approaches such as the conventional fixed contingency approach. From a theory development perspective, it is important to validate the theoretical framework underpinning the RCF approach or its variations through a comparison of predictive accuracy with historic accuracy data and other dominant approaches. For practitioners, such evidence will help them in choosing and adopting the best-practice project cost estimation method. This study compares a sample using a hybrid estimating approach, blending RCF with fixed contingency approach with historic accuracy data and samples of two other dominant estimation methods, namely the conventional fixed contingency approach and RBE. The results are presented and discussed.
In the sections below, literature is reviewed. Then, data collection is described, sample characteristics outlined and the analysis process explained. Third, the results are presented; validity threats, implications and future directions are discussed. Finally, conclusions are drawn.
This section reviews relevant literature on estimation methods. It starts with cost estimation accuracy at various project stages. Then, past findings on cost overruns of infrastructure projects and factors causing the cost overruns are reviewed. Finally, the three most commonly used cost estimating procedures are introduced and research question specified.
A typical construction project can be divided into five stages, conceptual or strategic stage, conceptual design stage, tender or detailed stage, preconstruction stage, and build stage (Liu and Zhu, 2007). At each of these stages, the amount of money required to undertake a project is estimated. Both the level of organizational resources committed to cost estimation, and the estimation accuracy, increase as a project progresses from early to later stages. A typical project cost estimate includes a base estimate, which accounts for physical material and labor quantities, plus a risk contingency, which quantifies the level of uncertainty or risk associated with the estimate. Figure 1 shows the two components that constitute project cost estimate.
The base estimate is generally derived through the multiplication of dollar rates and quantities of project components. Risk contingency “is expected to be expended [and is] an amount added to an estimate to allow for items, conditions, or events for which the state, occurrence, or effect is uncertain and that experience shows will likely result, in aggregate, in additional costs” (Anonymous, 2007). There are various methods that can be used to quantify the risk contingency, such as reference class forecasting, the conventional contingency approach and risk-based estimating, each of which is introduced below.
The cost estimates produced at different stages of the same project carries different levels of uncertainties/risks and thus different estimation accuracies. As a project progresses through its lifecycle, more information about the project's scope, design, and specifications become available, which enable the estimation team to more accurately estimate the quantity and price of material and resources. As a result, generally more risk contingency is applied at the earlier stages than in later stages. Studies have shown that the typical estimation accuracy is 30% to 50% at the conceptual stage (Ashworth & Skitmore, 1982); around 20% at the design stage (Morrison, 1984); about 10% at the tendering stage (Flanagan & Norman, 1983), and; 5% during preconstruction stage (Ferry & Brandon, 1991).
Most studies examine estimation accuracy by comparing the estimate at the tender stage with the actual cost of the projects. Because it is generally prepared after tendering and influenced by the tender price, preconstruction cost estimate is primarily used for cost control purposes and not used to assess project estimation accuracy.
Project cost performance and causal factors for cost overruns
Despite decades of efforts in reducing project cost overruns, large infrastructure projects still continue to be plagued by delays and large cost overruns. The collective evidence from studies on project cost performance conducted throughout the world shows the recurring pervasiveness of cost overruns (Flyvbjerg et al., 2002; Pickrell, 1990; Merewitz, 1973). Further, the phenomenon is not limited to any geographic location, impacting on infrastructure projects all around the world (Flyvbjerg et al., 2005b).
An early study conducted in the 1980s by the Department of Industry, Technology and Resources, Victoria (DITR, 1989), Australia, examined project performance of 21 infrastructure projects from both the UK and Australia with a total value of $7.66 billion (in 1988 Australian dollar). Of the 21 projects, six were delivered by the public sector and the remaining 15 were conducted by the private sector. The study found that 50% of projects delivered by government entities exceeded their estimated cost by more than 20%. In contrast, cost performance by the private sector was better with 60% of the projects having actual costs within ±5% of estimates.
Table 1 compiles the key statistics by studies examining the cost performance of various infrastructure projects across the world. Table 1 shows that, of the 2,221 projects investigated, the average overrun for each study ranges from 5% to 88% with a mean of 34.7% while the standard deviation ranges from 29.2% to 62% with a mean of 37.8%. These statistics shows that the cost overrun for infrastructure projects is substantial and that the consistency of cost estimates are far from satisfactory. To quote from one of the most cited studies, Flyvbjerg et al. (2005b) found that cost estimates for large infrastructure projects are “highly and systematically misleading”. Their study found that 86% of all examined projects (258 in total) exceeded their estimated cost with an average overrun of between 20% and 45%.
|Author & Year||Sample Size||Type of Project||Mean||SD|
|Odeck, 2004||610||Road Infrastructure||7.88%||29.20%|
|Odeck & Skjeseth, 1995||12||Toll Roads||5%||--|
|Bertisen & Davis, 2008||63||Mining and Smelting Projects||25%||30%|
|Flyvbjerg, Holm, & Buhl, 2004||58||Rail||45%||38%|
|Flyvbjerg, Holm, & Buhl, 2004||33||Bridges and Tunnels||34%||62%|
|Flyvbjerg, Holm, & Buhl, 2004||167||Roads||20%||30%|
|Pohl & Mihaljek, 2002||1015||World Bank Projects||22%||--|
|Fouracre, Allport, & Thomson, 1990||21||Metro Projects||45%||--|
The above studies have identified a number of factors contributing to persistent cost overrun. Poorly defined scope or significant scope changes that were not accounted in the base estimate could lead to significant overruns (Picckrell, 1990; Bertisen & Davis, 2008). Similarly, incorrect quantities and unit rates for material and labor quantities could produce inadequate bases estimate and subsequently lead to overruns (Odeck & Skjeseth, 1995; Pohl & Mihaljek, 1992).
Some scholars have argued that the cost estimates of projects are sometimes so complex and unique that the ability to learn from past mistakes is limited (McMillan, 1992). Further, Touran and Lopez (2006) contend that given the time length between estimation and project completion, price fluctuation in construction costs tends to increase the difficulty of estimating and drives estimates to become very unreliable. Whilst acknowledging that both poor scope definition and mistakes within the preparation of the base estimate do lead to error, a number of scholars (Odeck, 2004; Flyvbjerg et al., 2005b; Bertisen & Davis, 2008) have argued strongly that the scope changes and errors in preparing the estimates should not lead to persistent cost overruns. Instead, they argue that the estimating errors derived from scope changes and preparation errors should be random and unsystematic, thus in the long term should not result in a systematic bias (e.g. either persistent under run or overrun). The errors could also come from imperfect techniques, inadequate data, honest mistakes, modeling errors and lack of experience (Bruzelius et al., 2002; Morris & Hough, 1987; Wachs, 1990). Mistakes happen, however the human ability to learn from mistakes suggests that the extent of these errors should improve over time (Flyvbjerg et al., 2002).
“Optimism Bias” has been identified as another primary cause of underestimation (Flyvbjerg, 2006; Kahneman & Lovallo, 1993; Mackie & Preston, 1998). The term was coined by Kahneman and refers to the human tendency to exaggerate one's own ability and underestimate potential difficulties (Lovallo & Kahneman, 2003; Kahneman & Tversky, 1979). It deals with an unintentional cognitive predisposition to view a situation more positively than is warranted (Flyvbjerg, 2006; Kahneman & Lovallo, 1993; Kahneman, 1994; Kahneman & Tversky, 1979).
People tend to be optimistic about what they can achieve and this optimism can lead to ‘planning fallacy’ (Lovallo & Kahneman, 2003). A project usually starts with a preliminary plan, which typically puts the project in a positive light. The subsequent analysis of the project is anchored on the preliminary plan and is effectively skewed toward over optimism (Lovallo & Kahneman, 2003). This anchoring is one of the main reasons that cause systematic bias in cost estimation. The problem becomes particularly acute when pessimistic views are suppressed and optimistic opinions rewarded and, as a result, people lose the ability to think critically (Lovallo & Kahneman, 2003).
Finally, strategic misrepresentation, which refers to the intentional deception designed to secure a favorable outcome for the project promoters (Flyvbjerg et al., 2002), has been identified as a major contributor to persistent project cost overrun. For example, contractors are motivated to secure contracts by lowering bidding price if they can jack up prices through variations during the construction phase. Trujillo et al. (2002) conclude that politicians have an incentive to embark on construction of large infrastructure projects because they enjoy a greater gain from such announcements compared to the losses incurred by having to raise taxes later. The Sydney Opera House is an example of strategic misrepresentation. With an actual cost 15 times the original, promoters knew that such an expenditure would be seen as unfavorable in the public eye and strategically underestimated it to ensure that prompt approval for the construction was granted (Flyvbjerg, 2005).
Cost forecasting methods and their relative estimation accuracies
The dominant method used in the construction industry can be classified into three types: the reference class forecasting (RCF), conventional contingency approach and RBE. Below, the three types of methods are described and propositions about their relative accuracy explained.
The conventional contingency approach
There are various techniques that can be used to estimate risk contingency including subjective judgment, sensitivity analysis, real options analysis, and Monte Carlo simulation (Akintoye & MacLeod, 1997). The most commonly used method is the deterministic contingency application (hereafter referred to as Conventional Contingency Approach) (Tummala et al., 1997). In Australia, construction contractors generally use such deterministic models to assess risks and opportunities in projects (Fayek et al., 1998).
The deterministic contingency approach adds an arbitrary percentage to the base estimate to account for risks. The contingency amount is typically determined based on an estimator's intuition or past experience. The contingency percentage depends on the type and lifecycle phase of the project with little consideration for the component risks involved (DITR, 1989). For example, a company may apply 10% contingency to the project estimate of all building construction projects and 20% to all civil engineering projects, regardless of differences in risks in the project cost components such as machinery or financing costs. Because of the arbitrary nature of this approach, it is especially vulnerable to optimism bias and strategic misrepresentation.
Reference class forecasting
The RCF approach mitigates optimism bias and strategic representation by taking the “outside view”. It uses a database of actual performance of comparable past projects within a given reference class to provide an objective reference point for the cost forecast of a current project (Flyvbjerg et al., 2005a). Through Monte Carlo simulations, the distribution of project cost overrun in the class is derived. One commonly used simulation result is the so-called S-curve, which visually links the probability of overrun with specific percentage project budget overrun. Based on the project estimators’ preference for risk overrun and the S-curve, an uplift percentage is determined and added to the base estimate as risk contingency (Flyvbjerg, 2006).
RCF is starting to be adopted by the construction industry including the British Department for Transport and a State Road and Traffic Authority in Australia (hereafter referred to as the Authority). Since it is a relatively new practice, few extant studies have examined its effectiveness. Flyvbjerg and COWI (2004) are the first to provide a comprehensive guideline for using RCF to mitigate the effects of optimism bias. In practice, there is no “standard” RCF code of practice. Variations exist on how the approach is applied.
The main challenge for applying the RCF method is the accumulation of a sample of similar projects with large enough sample size and accurate cost information. It may take a very long time to develop such a database. For some types of projects that are relatively rare in a country (e.g. nuclear power plants or large-scale desalination plants), it may never be possible to have a sample size large enough for statistical analysis. The problem is further exacerbated by the fact that private companies may not be willing to share such commercially sensitive information with competitors or the governmental agencies.
RBE takes a step further in modeling project costs. Rather than add a contingency to the total project cost as in RCF and the conventional contingency approach, RBE models the cost of individual components with base estimates and stochastic risk contingencies. The distribution of the overall project cost (or project budget S-curve) is derived by summing up the stochastic cost components (Shaheen et al., 2007). The S-curve relates specific project budgets with corresponding probability of cost overrun. The project cost estimate is determined using the S-curve based on an organization's estimation procedure and the estimating team's preference for risk of cost overrun.
RBE method starts with the identification of two categories of risks: inherent risks and contingent risks. Inherent risks are those that are internal to a project and uncertain in quantities, rates, and scope such as material or labor costs. Contingent risks are external events that may occur throughout the life of a project but not assumed in the base estimate (Aspinall & Trueman, 2006). For example, the risk of an earthquake during construction. The two types of risks are handled differently in that contingent risks are assigned a probability of occurring (usually low probability events) while inherent risks are assumed to exist throughout the construction process (Uher, 1996). For details about how the risks are quantified in RBE, please refer to Napier and Liu (2008).
Instead of relying on the simulation results of past project performance data, the RBE approach can take into account outside views from a variety of sources, both the subjective judgment from different experts and stakeholders, and objective data if available. The typical process is to go through a risk workshop attended by experts, stakeholders, and estimators, during which time the base estimates and corresponding risk contingencies for each individual components are determined through a consensus-building process (Aspinall & Trueman, 2006). Then, Monte Carlo simulation is conducted for the total project budget (Napier & Liu, 2008; Chau, 1995).
RBE has only recently been introduced into Australia. The study by Napier & Liu (2008) examined estimation accuracy of 11 recent projects undertaken by the Sydney Water Corporation. Preliminary findings suggest that RBE has good predictive validity with 10 of the 11 projects had an actual cost within the range of the risk-based estimate.
In practice, organizations tend to tailor generic estimating approaches to their unique operating environments by developing hybrid approaches blending various approaches. For example, rather than conducting Monte Carlo simulation analysis for each project to decide the amount of uplift needed for each project as recommended by Flyvbjerg and COWI (2004), some organizations choose to simplify the process by working out a range for the uplift and then apply it to project components as ranges for contingencies. Another common hybrid practice is to require all the estimates be checked against accumulated historical data on similar estimates—reference to “outside view”.
Despite the increasing popularity of the RCF approach and its variations, there is very little empirical evidence on the accuracy of RCF or its variations, which is the motivation for this study. The research question for this study is: How accurate is the hybrid estimating approaches?
In this section, the data collection process and sample characteristics are outlined first. Then the analysis process is explained.
RCF sample data
The primary data sample used in the study comprise of 52 road projects during 2000 and 2005 by the Authority, of which, the data on 44 projects were collected directly from the Authority with a total value of $2.17 billion and the remaining eight are collected from the auditing report by the Office of the Auditor General, Australia. Because all the projects in the sample were estimated following a procedure that blends RCF with fixed contingency approach, we refer to this sample as the hybrid sample.
The Authority's estimating manual requires that contingency to project estimates be made within specified ranges depending on the stage and type of the estimate. For example, contingency for detailed estimates needs to be within the range of 10% to 20% while 20% to 35% for concept estimate stage. Contingencies outside the ranges specified in the manual need to be justified. The contingency ranges are determined based on historical performance data—a practice that mirrors the RCF approach. The manual also stipulates that estimates be checked for discrepancy against historical data accumulated over time by the Authority adjusted for inflation and time—another practice at the core of RCF, seeking objective, outside views.
The data was compiled based on past project reports. The data contains estimates at three different stages: strategic, concept, and detailed, respectively. Only the detailed stage estimates are used in this paper. In addition, the actual cost for each project was also provided. Detailed estimates were provided for 30 of the 44 projects. All of the estimates were conducted following the 2001 version of the Authority's project estimating manual.
Since any significant scope change in a project could account for the escalation of costs, it is important to partial out the scope change effect to ascertain the accuracy of specific estimation methods. Discussions with Authority revealed that two of the projects had significant scope changes and the remaining 42 projects did not experience any significant scope changes. Subsequently, we have excluded the two projects with significant scope changes from the sample.
Data on a further eight completed Authority projects with a total value of $1.57 billion were obtained from the Auditor General's Report to Parliament (AGNSW, 2007). The estimate and final cost for each project is given in June 2007 dollars. Subsequent discussions with the Authority indicate that these projects were also estimated using the 2001 project estimating manual—using the hybrid approach. Combined with the 42 Authority projects, the final sample contained data on 50 projects.
The estimates and actual costs were produced at different years. Most estimates have been provided dates about when the estimation was conducted. For those estimates without specific dates, the Authority suggests that there is generally two years between the detailed estimate and actual cost. To account for differences in time value of money, the estimates and actual costs have all been converted to June 2008 dollars using the Road and Bridge Construction Index (RBC) as the relevant inflation rate. The RBC takes into consideration prices of material and labor commonly used in infrastructure construction projects and hence is considered an appropriate measure for cost escalation experienced by projects in this study.
The index is compiled quarterly each year. The reference dates provided with the data do not necessarily match with the index compilation dates. In this study, the RBC value for a particular date, which falls between the two compilation dates, is determined by linearly interpolating the index between the closest compilation dates to give a weighted average index at the reference date.
The conventional project sample
The RCF performance data was compared with the estimating performance of a sample of Australian construction projects using the conventional fixed contingency approach. The sample was collected by first identifying 232 Australian construction companies, including general, heavy construction, and special trade contractors within the definition of the Standard Industry Classification (SIC) code, 15211799, with annual turnover exceeding A$40 million (Australian dollar).
A fax was sent to each Chief Executive Officer (CEO) or Managing Director seeking a list of project managers who could potentially participate in the survey. Questionnaires were sent to the 200 project managers identified, of which 41 completed and returned the questionnaire, representing a response rate of 20 percent. The project size ranged from A$1.2million to A$191million with a mean of about A$31million.
The respondents were asked to answer in percentage terms the deviation of actual project cost from estimated cost. As in the RCF sample, significant scope changes could seriously affect the accuracy as measured in this study and thus the validity of the findings. To guard against the risk, we had asked the respondents to indicate “if the project was substantially re-scoped”. The projects that have been significantly re-scoped had been excluded from the analysis. In addition, to partial out the effects of small adjustments to project costs on accuracy, the respondents were instructed to use adjusted cost as the estimated cost “if project budget were adjusted during the course of the project by agreement with the client”.
The survey did not specifically identify the estimation approach. However, it is unlikely that RCF was adopted by any Australian organization before this survey (1999-2000). Given that all projects in the sample had already been completed by the time of the survey, it is most likely that all project in this sample used the conventional estimating method.
The RBE sample
The RBE sample data was based on data reported by Napier and Liu (2008). The sample contains 11 water infrastructure projects by the Sydney Water Corporation (SWC) and the Sydney Catchment Authority (SCA) in the state of NSW, Australia. All the estimates were performed by the same estimating team between 2003 and 2007. The use of the same estimating team ensures consistency in the implementation of RBE, thus reducing the possibility that differences in implementation could affect the findings. Projects with numerous and significant scope changes were excluded.
The projects using the RBE method included in this sample adopted two types of contracts—Design and Construct (D&C) and Alliance contract. Since risks are treated differently in D&C and Alliance types, the measure of estimation accuracy had to be adjusted to each project type. In summary, the accuracy in alliance projects is determined using either the total cost estimate (TCE) or budget curves, while the accuracy in D&C projects is evaluated using the budget curve. For details about the adjustments, please refer to Napier and Liu (2008).
Drawing from the literature on forecasting accuracy (Holden & Peel, 1988; Makridakis & Hibon, 1984; Fitzgerald & Akintoye, 1995), the cost estimation accuracy is measured in two dimensions—the estimation error and the degree of dispersion in the sample around the mean. The accuracy of the hybrid sample is compared with historical data. Further, the accuracy in the hybrid sample is contrasted with accuracies in the conventional and RBE samples, respectively, to explore potential relative differences in accuracy.
The first dimension of accuracy was measured as the percentage difference between the detailed estimate and the actual cost. This is a good indicator for the accuracy of cost estimation for a specific project. However, our interest here is to find out the accuracy of the hybrid approach as a method for cost estimation. A typical approach is to measure the degree of dispersion around the mean estimation error for projects using the hybrid approach (RTA, 2008; AGNSW, 2007; Holden & Peel, 1988; Makridakis & Hibon, 1984). In this study, the second dimension—the degree of dispersion in the sample around the mean—is measured by the standard deviation in the sample. The two dimensions need to be used in conjunction to measure accuracy of an estimating method. The first dimension measures the closeness of the estimate to the final actual cost and the second dimension measures the consistency of the estimation method.
The contrast of accuracies across the samples is conducted using the independent t-test procedure of SPSS, which simultaneously test the equality of means (using t-test for equality of means) and variances (using Levene's test for equality of variance). A significant test result (p<=0.05 for two-tailed tests) indicates the existence of statistically significant difference (a difference greater than zero).
To further aid the analysis, log-normal distribution curves of the frequency histogram of the estimation errors in each of the samples are overlaid in Figure 2 to visually demonstrate the contrast in accuracies across the two samples. Past studies have shown that log-normal distribution is appropriate for modeling cost estimation performance (Rothwell, 2005).
Results and Discussions
The results are presented in three steps. First, comparison of the hybrid accuracy with historical results and accuracies in the RBE and conventional samples are reported. Second, the implications for practice and theory development are discussed. Third, validity threats and limitations are discussed. Finally, conclusions are drawn and further studies suggested.
Accuracy of the hybrid approach
Table 2 reports the mean and standard deviation of the hybrid sample. Compared to historical average mean of 34.7% and standard deviation of 37.8% (See Table 1), a mean of -9.71% and SD of 14.48% for the hybrid sample represents remarkable improvement.
The test results for equality of means and variances are presented in Table 3. It shows that the means of estimation errors in the hybrid and the conventional samples, respectively, are significantly different (p<=0.00). Considering the negative mean estimation error (under budget) in the hybrid sample and the positive mean estimation error in the conventional sample (-9.71% vs. 3.88%, See Table 2), the results suggest that the mean estimation error of using the hybrid approach is lower than that using the conventional contingency approach.
Table 3 further reports that the variances of estimation errors in the hybrid sample and the conventional samples, respectively, are not significantly different (F=2.97, n.s.).
On the relative accuracy between the hybrid and RBE approaches, the results in Table 3 suggest that the latter is more accurate. Table 3 shows that the means of estimation errors in the hybrid and the RBE samples, respectively, are significantly different (p<=0.05). The higher absolute mean estimation error in the hybrid sample (9.71%, See Table 2) supports the hypothesis. Further, Table 3 shows that the variances of estimation errors in the hybrid and the RBE samples, respectively, are significantly different (F=4.83, p<=0.03). Considering the variance of estimation error in the hybrid sample is higher than that in the RBE sample (See Table 2), the results indicate higher variance in the hybrid sample than in the RBE sample. Therefore, the results here suggest that the RBE approach could be more accurate than the hybrid approach.
|Sample||N||Mean||Std. Deviation||Std. Error Mean|
* a negative value denotes under-budget while a positive figure indicates over-budget
|Levene's Test for Equality of Variances||t-test for Equality of Means|
|F||Sig.||t||df||Sig. (2-tailed)||Mean Difference|
|Hybrid vs. RBE||4.83||0.03||2.07||40.78||0.05||6.22|
|Hybrid vs. Conventional||2.97||n.s.||4.66||62.17||0.00||13.59|
The results reported above show that the hybrid approach adopted by the Authority represents significant improvement against historical averages on infrastructure projects. Comparison with the conventional contingency approach, respectively, shows that the hybrid approach can produce more conservative estimates, either by less amount over-budget or, more likely, under-budget than (and have similar dispersion around the mean estimation error with) the conventional contingency approach. In contrast, comparison with a small RBE sample provides the preliminary evidence that RBE can be more accurate than the hybrid approach.
Validity threats and future research directions
Scope creep and significant scope changes are typically not included in the original estimate but could have significant effects on the actual costs, thus affecting the accuracy. We have mitigated this risk by excluding projects with considerable scope change in the hybrid and the conventional samples. For the RBE sample, adjustments have been made to the costs to account for significant scope changes (Napier & Liu, 2008).
The small sample size in the RBE sample could lead to type II error, i.e. not being able to detect the existing differences in hypothesized relationships. The only hypothesis rejected relates to the difference in variance of estimation error in the two samples—the hybrid and the conventional sample. To rule out the possibility, the hypothesis needs to be further tested in future studies using larger size samples.
Finally, the hybrid sample only includes road and traffic projects conducted by one organization. Similarly, the RBE sample contains only water infrastructure projects. It thus poses a validity threat to the generalizability of the findings to non-road projects using similar hybrid or non-water projects using RBE or projects by organizations other than the Authority. The question of generalizability can only be answered by future studies examining accuracies of various methods in different types of projects. In particular, random sampling of different project types from a variety of organizations using a specific estimating approach should be strongly encouraged. Nevertheless, the results from this study provide preliminary insight on the relative accuracy among the three estimation approaches, which should be followed up by subsequent studies.
The findings contribute to the theory development by providing empirical evidence on the relative accuracy of the three methods in cost estimation. The findings largely support the view that there is estimation bias toward over-budgeting, which can be mitigated by taking the outside view. More importantly, modeling individual project cost components as statistical distributions (as in RBE) does appear to significantly improve the consistency of estimates. However, due to limitations in the randomness of the sample and the types of projects and organizations included, the findings need to be interpreted with caution and further validation on different types of projects by a variety of organizations is warranted.
It is important to note the differences in results from previous studies. In the two largest studies conducted by Flyvbjerg et al. (2005b) and Odeck (2004), the estimation errors between the detailed estimation and the actual cost have standard deviations of 30% and 29.2%, respectively, the lowest of published studies on infrastructure projects. In comparison, the hybrid sample exhibits a much narrower dispersion with a standard deviation of only 14.48%. It is not clear whether the difference in dispersion is due to factors unique to the organizations (e.g. the Authority) or due to the fact that the Flyvbjerg et al. (2005b) and Odeck (2004) samples may include a mixture of projects using different estimating approaches. Studies on the accuracy of specific estimation approaches rather than a mixture of approaches would be a more fruitful path in developing theories and practices for effective estimation.
In contrast to the prevalence of optimism bias (Bruzelius et al., 2002; Flyvbjerg et al., 2005a), the effect of adding an uplift to estimates is the increased likelihood of being under-budget (75% of the projects in the sample are not over-budget). In other words, the result shows a bias toward being under-budget. Although under-budget is typically preferred to over-budget, large amount of under-runs or persistent under-budget could lead to lost opportunities that would otherwise have been undertaken if the estimate had been more accurate. Figure 2 shows that, supporting the findings, mean estimation error of the hybrid tends to be biased toward under-budget and has a wider dispersion compared to that of RBE. In contrast, RBE has the least dispersion of estimation error and produces the most accurate estimates. Further studies should investigate the contributing factors to hybrid's bias toward under-budget. For practitioners, continuous monitoring and fine-tuning of estimation procedures while understanding the under-budgeting possibility of hybrid approach is critical to ensure the effective implementation of the RCF approach.
Supporting the argument that RCF mitigates against optimism bias and strategic misrepresentation, this study finds that a hybrid approach, which blends components of RCF and the conventional contingency approach, produces estimation accuracy significantly better than historical results reported in literature. Comparison with a conventional projects sample provides further support for the validity of the hybrid approach, which shows the hybrid approach is more accurate than the conventional contingency approach in that the hybrid approach tends to produce estimates that are more likely to lead to under-budget rather than over-budget. However, the dispersion around the mean estimation error of using the hybrid approach is similar to that using the conventional contingency approach.
The comparison with the RBE sample suggests that the RBE approach could be more accurate— closer to the mean estimate and narrow dispersion. This finding is consistent with the view that detailed modeling of individual components produces more accurate overall estimation than either ignoring distributional information (as in conventional approach) or simply modeling the distribution of overall project cost (as in RCF).
The finding also highlights the tendency of the hybrid approach to be overestimated. It has important implications for both practice and future research.
Li Liu, PhD, M.Tax, MBA, BE
Dr. Liu is currently a senior lecturer in project management at the University of Sydney. He has a doctorate degree from the Australian Graduate School of Management (AGSM). He also has a master's in Taxation (Usyd), a MBA (AIT) and a BE (NUTD).
Dr. Liu has one decade of working experience in systems engineering, project management, business research, management consulting, and managing an E-com company. His research interests include management of infrastructure projects, enterprise-level project management, information technology/information systems project management, organizational control theory, programme/portfolio management, and project/programme governance.
The courses he has taught include Managing Project-Oriented Organization, Project Portfolio and Programme Management, Strategic Project Management, Project Formulation, IT project management, and Business Information Systems.
He is published in Institute of Electronics and Electrical Engineers (IEEE) Transactions on Engineering Management, Project Management Journal, Journal of Information Technology (JIT), Construction Management & Economics, and other international Journals, Institut Européen d'Administration des Affaires (INSEAD), and a number of international conferences including PMI Research Conference and International Conference for Information Systems (ICIS). He served as a reviewer for a number of journals and conferences including Management Information Systems Quarterly (MISQ), JIT, ICIS, ECIS and PMI Research Conference. He also served as an associate editor for ICIS'09. Currently, he serves on the editorial board of JIT.
George Wehbe, BE, BCom
George Wehbe is a graduate from the University of Sydney and holds a Bachelor of Engineering (Project Engineering Management) with 1st Class Honours and a Bachelor of Commerce (Business Information Systems). His honours thesis looked into the effect of different risk quantifications methods on the accuracy of cost estimates in large infrastructure projects in an Australian State. Mr. Wehbe currently works as a Business and Systems Development Manager within his own Information Technology (IT) company and employs more than 20 staff. He has interests in IT, Business Development, Cost Engineering, Business System Analysis, and Development.
Jonathan Sisovic, BE, BSc
Jonathan Sisovic graduated from the University of Sydney with a Bachelor of Engineering (Civil) (Hons) and Bachelor of Science in Physics. His thesis topic looked at the cost estimation practices of the construction industry and their effectiveness. Mr. Sisovic currently works as a Project Engineer for an infrastructure construction company, Abergeldie Complex Infrastructure. His interests include project management, cycling, 60s music, and astronomy. He currently resides in Sydney, Australia.
AGNSW (2007). Auditor-General's Report to Parliament 2007 Volume 6, Auditor General of New South Wales, Sydney, New South Wales, Australia. [Online]. Available: http://www.audit.nsw.gov.au/publications/reports/financial/2007/vol6/pdf/31_0342_roads_and_traffic_authority_of_new_south_wales.pdf
AGS (1994). Riksrevisionsverket. Infrastrukturinvesteringar: En kostnadsjamorelse mellan plan og utyfall I 15 storre prosjekt innom Vagverket og Banverket. Vol 23, Auditor General of Sweden, Sweden.
Akintoye, A. S., & MacLeod, M. J. (1997). Risk analysis and management in construction. International Journal of Project Management, 15(1), 31-38.
Anonymous (2007). Cost Engineering Terminology, AACE International Recommended Practice No. 10s-90, AACE International.
Ashworth, A., & Skitmore, M. (1982). Accuracy in estimating. Chartered Institute of Building -occasional paper, no. 27, London.
Aspinall, A., & Trueman, P. (2006). Optimising Asset Management Decision Making and Budgeting Using Risk Management Techniques. Engineering Asset Management Proceedings of the 1st World Congress on Engineering Asset Management (WCEAM), 11–14 July 2006, pp 229-236.
Bennet, R. (1997). Technical due diligence requirements for mining project finance. Presented at the 85th Annual Global Mining Opportunities and 2nd Annual Copper Hydromet Roundtable, Vancouver, British Columbia, 1997.
Bertisen, D., & Davis, G (2008). Bias and error in mine project capital cost estimation. The Engineering Economist, 53(2), 118-139.
Bruzelius, N., Flyvbjerg, B., & Rothengatter, W. (2002). Big decisions, big risks, improving accountability in megaprojects. Transportation Policy, 9(2), 143-154.
Castle, G. (1985). Feasibility studies and other pre-project estimates: how reliable are they? New York: Society of Mining Engineers AIME.
Chau, K. W. (1995). The validity of the triangular distribution in Monte Carlo simulation of construction costs. Construction Management & Economics, 13(1), 15-21.
DITR (1989) The Predictability of Australian Project Performance, Canberra, Australia: Department of Industry, Technology, and Resources.
Fayek, A., Young, D. M., & Duffield, C. F. (1998). A survey of tendering practices in the Australian construction industry. Engineering Management Journal, 10(4), 29-34.
Ferry, D. J., & Brandon, P. S. (1991). Cost Planning of Building, Oxford U.K.: BSP Professional Books.
Fitzgerald, E., & Akintoye, A. (1995). The accuracy and optimal linear correction of UK construction tender price index forecasts. Construction Management Economics, 13, 493-500.
Flanagan, D. J., & Norman, G. (1983). The accuracy and monitoring of quantity surveyor's price forecasting for building work. Construction Management Economics, 1(2), 157-180.
Flyvbjerg, B. (2005). Design by deception: the politics of megaproject approval. Harvard Design Magazine, Spring/Summer (22), 50-59.
Flyvbjerg, B. (2006). From Nobel Prize to project management: getting risks right. Project Management Journal, 37(3), 5-15.
Flyvbjerg, B., Bruzelius, N., & Rothengatter, W. (2003), Megaprojects and Risk: An Anatomy of Ambition. Cambridge, UK: Cambridge University Press, 2003.
Flyvbjerg, B., & COWI (2004). Procedures for dealing with optimism bias in transport planning, London, UK: British Department of Transport, 2004.
Flyvbjerg, B., Holm, M. S., & Buhl, S. (2002). Understanding costs in public works projects: errors or lie? Journal of American Planning Association, 68(3), 279-295.
Flyvbjerg, B., Holm, M. S., & Buhl, S. (2005a). How (in)accurate are demand forecasts in public works projects? Journal of American Planning Association, 71(3), 131-146.
Flyvbjerg, B., Holm, M. K., & Buhl, S.L. (2005b). What causes cost overrun in transport infrastructure projects? Transport Reviews, vol. 24(1), 3-18.
Gypton, C. (2002). How have we done? Feasibility performance since 1980. Engineering and Mining Journal, 302(1), 41-46.
Holden, K., & Peel, D. (1988). A comparison of some inflation, growth and unemployment forecasts. Journal of Economic Studies, 15, 45-52.
Kahneman, D. (1994). New challenges to the rationality assumption. Journal of Institutional and Theoretical Economic Studies, 150, 18-36.
Kahneman, D. & Lovallo, D. (1993). Timid choices and bold forecasts: a cognitive perspective on risk taking. Management Science, 39(1).
Kahneman, D., & Tversky, A. (1979). Prospect theory: an analysis of decisions under risk. Econometrica, 47, 313-327.
Liu, L., & Zhu, K. (2007). Improving cost estimates of construction projects using phased cost factors. Journal of Construction Engineering & Management, 133(1), 91-95.
Lovallo, D., & Kahneman, D. (2003), Delusions of success: how optimism undermines executives’ decisions. Harvard Business Review, 81(7), 57-63.
Mackie, P., & Preston, J. (1998). Twenty-one sources of error and bias in transport project appraisal. Transportation Policy, 5(1), 1-7.
Makridakis, S., & Hibon, M. (1984). “Accuracy of forecasting: an empirical investigation” in The Forecasting Accuracy of Major Time Series Methods, S. Makridakis, Ed. New York: John Wiley and Sons.
McMillan, J. (1992). Games, Strategies & Managers, New York: Oxford University Press.
Merewitz, L. (1973). “Cost overruns in public works” in Benefit Cost and Policy Analysis, Chicago, Aldine, pp. 277-295.
Merrow, E. W., McDonnel, L. M., & Arguden, R. Y. (1988). Understanding the outcomes of megaprojects: a quantitative analysis of very large civilian projects, Santa Monica, CA: The Rand Corporation
Morrison, N. (1984). The accuracy of quantity surveyors cost estimating. Construction Management & Economics, 2(1), 57-75.
Morris, P. W., & Hough, G. H. (1987). The anatomy of major projects: a study of the reality of project management, New York: John Wiley and Sons.
Napier, Z., & Liu, L. (2008). The predictive validity and performance drivers of risk-based estimating: case studies of Australian water infrastructure projects. Proceedings of Project Management Institute Research Conference, [CD-ROM], Warsaw, Poland, 2008.
Odeck, J. (2004). Cost overruns in road construction–what are their sizes and determinants. Transportation Policy, 11(1), 43-53.
Odeck, J., & Skjeseth, T. (1995). Assessing Norwegian toll roads. Transportation Quarterly, 49(2), 89-98.
Pickrell, D. H. (1990). Urban rail transit projects: forecast versus actual ridership and cost, Washington, DC: U.S. Department of Transportation.
Pickrell, D. H. (1992). A desire named streetcar: Fantasy and fact in rail transit planning. Journal of the American Planning Association, 58(2), 158-176.
Pohl, G., & Mihaljek, D. (1992). Project evaluation and uncertainty in practice: a statistical analysis of rate-of-return divergences of 1015 World Bank Projects. World Bank Economic Review, 6(2), 253-277.
Rothwell, G. (2005). Cost contingency as the standard deviation of the cost estimate for cost engineering. Cost Engineering, 47(7), 22-25.
Shaheen, A. A., Fayek, A.R., & AbouRizk, S.M. (2007). Fuzzy numbers in cost range estimating. Journal of Cost Engineering Management, 133(4), 325-334.
Thomas, S. (2001). Project development costs–estimates versus reality. Proceedings of the 10th Annual Conference of the Mineral Economics and Management Society, April 4-6, 2001.
Touran, A., & Lopez, R. (2006). Modelling cost-escalation in large infrastructure projects. Journal of Construction Engineering & Management, 132(8), 853-860.
Trujillo, L., Quinet, E., & Estache, A. (2002). Dealing with demand forecasting games in transport privatization. Transportation Policy, 9, 325-334.
Tummala, V. M. R., Leung, H. M., Mok, C. K., Burchett, J. F., & Leung, Y. H. (1997). Practices, barriers and benefits of using risk management approaches in selected Hong Kong industries. International Journal of Project Management, 15(5), 297-312.
Uher, T. E. (1996). A probabilistic cost estimating model. Cost Engineering, 38(4), 33-40.
Wachs, M. (1990). Ethics and Advocacy in forecasting for public policy. Business and Professional Ethics Journal, 9(2), 141-157.
© 2010 Project Management Institute