Abstract
Confronted with the recurring problem of large cost overruns and late delivery of large public investment projects, the Norwegian government initiated a review of such projects and the planning system at the end of the last century. It was done to identify the main causes and suggest means to improve governance of such large investments. The aim was to avoid embarrassing cost overruns in the future, but also the type of flawed investment cases one had seen in the past, and improve return on investments. As the result of the study, a quality assurance scheme was established year 2000 under the Ministry of Finance to help strengthen overall governance of large public projects.
Megaprojects take many years to prepare and implement. Only today, 13 years after, a sufficiently large number of projects had been completed to allow researchers to document some of the effects of the scheme and its wider impact on government and industry. The results, some of which are very encouraging, are presented and discussed in this paper.
Keywords: project governance; quality assurance; cost estimation
Measuring Project Success
Megaprojects are recognized as an important part of corporate and public life, but with varying reputations. This is reflected in the media where unsuccessful cases seem to get more publicity than the successful ones. The most common type of failure the media appears to report on is cost overrun. Number two is schedule overrun. That is because information on cost and progress is more easily available than more complex issues regarding relevance, etc. Even if the outcome is seen to be useful, the media tend to focus essentially on the large public construction projects that have suffered huge cost or time overruns, such as Denver’s US$5 billion airport 200% overspend (Szyliowicz & Goetz, 1995), or the U.K.’s Scottish Parliament coming in “10 times over budget and more than three years late” (Tempest, 2004).
Measuring success in a megaproject is not simple and straightforward. One reason is that success may be interpreted differently by different individuals and institutions, depending on their preferences, values, and to what degree they are affected by the project. Another is that the degree of success is time-dependent. For instance, Shenhar, Dvir, Levy, and Maltz (2001) offer a chronological sequence of events as a compound definition of project success: (1) Meeting time, budget, and other requirements, (2) impact on the customer, (3) benefit to the performing organization, and (4) preparing the future. The project’s stakeholders do not necessarily share the same view of success. The project manager typically sees his or her job successfully accomplished when the project is on time, within budget, and to specifications. The users will be concerned about the immediate effects of the project, and the investor or commissioner will typically be more concerned with the long-term economic viability. The user’s view on success is essential. Pinto (2007, p. 7) quotes from an Infoworld article describing “a U.S. Army study of IT projects [that] found that 47% were delivered to the customer but not used; 29% were paid for but not delivered; 19% were abandoned or reworked; 3% were used with minor changes; and only 2% were used as delivered.”
This is not to say that all megaprojects are managed badly—indeed, the management and governance of many megaprojects has made considerable improvements in recent years (see, for example, HM Treasury (2007) for an upbeat report on the U.K.’s Office of Government Commerce). After many decades with developments in the project management profession, there is more expertise now in delivering efficiently and successfully a well-defined pre-specified project within a well-defined constant environment. This has proved very valuable in certain circumstances. But the focus on project management has been much too narrow in the media and the public. Truly, a much wider view needs to be taken on the success and failure of projects. The initial choice of the megaproject concept is of critical importance. This represents the one key decision of many made during the lifetime of a megaproject, which is likely to have the largest impact on long-term success or failure (Williams, Samset, & Sunnevag, 2009). Here, by “the project concept,” we mean much more than just the technical solution—it includes the entire business case, the various organizations involved, and the various mechanisms and arrangements involved in the interorganizational relationships (see also Miller and Hobbs [2009]).
What is meant by successful projects can be understood in different ways. A common distinction is made between three levels of success:
- Operational success: The delivery of the project (the outputs) is as promised and is both time and cost efficient.
- Tactical success: The project produces the maximum utility/benefit for the users (outcome) at the lowest possible cost.
- Strategic success: The project contributes to desired societal development (as expressed by its long-term purpose), at the lowest possible cost and in a sustainable manner.
As indicated, the media often focuses mostly on operational success. This is because the project outputs can be verified immediately once the project is completed, and it is easy to benchmark cost efficiency against other projects. Tactical and strategic success can only be considered in the longer term and seen in relation to other measures and societal processes. Since major public investments typically have a broad societal perspective, the assessment of tactical and strategic performance will be vital aspects of the assessment of their success.
Cost Overrun in Large Projects
In project management, cost is commonly the management parameter that attracts the most attention both during the front-end phase and implementation. In many projects, however, even large cost overruns have had little effect on tactical and strategic project success. Yet in other cases, cost overruns may comprise a death blow.
Cost overruns in public as well as private projects have been a very common problem across countries and over time. The problem is widely discussed in academic literature on projects. A well-known study of large projects (Morris & Hough, 1991), examined more than 4,000 large government funded projects in defense, transportation, aviation, aerospace, energy, etc. The study found that overruns were typically between 40% and 200%, with overruns in the oil sector of up to 800% and in nuclear power plants of up to 4,000%. In the U.K., MacDonald (2002) showed that the final cost of 50 major projects was 24 - 36% higher than the budget. In another extensive study, Flyvbjerg, Skamris Holm, and Buhl (2003) analyzed 258 infrastructure projects in 20 countries over a period of 70 years. The conclusion was that the cost overruns were significant and that the cost estimates had not improved or worsened during the period. Nine out of ten projects had cost overruns with an average of 28%. Berechman and Wu (2006) studied 128 road projects in Vancouver, Canada, that began in the years 1993 to 2003 and found that as many as 104 of these had considerable cost overrun. Makovšek, Tominc, and Logožar (2012) found that a majority of Slovenian road projects had cost overruns 30% on average in projects begun in the period 1994–1999, and 19% for projects begun since 1999.
There are three common explanations why cost overruns occur:
- Technical problems, which are difficult to foresee due to insufficient information, experience, complexity, technologies, etc.
- Cognitive factors related to our limited ability to predict, optimistic bias, etc.
- Political explanations, such as low-cost estimates being accepted deliberately in order to increase the chance that the project will be accepted.
Flyvbjerg (2009) examines these various reasons behind the “pervasive misinformation” which persistently troubles project estimation. He suggests that in many cases, underestimation of cost is done on purpose (i.e., political explanations). Once “on the list,” the project stands a better chance to be implemented. Decision-makers demonstrate a surprising high degree of tolerance for what is accepted in terms of unrealistic estimates up front without repercussion for the involved parties. This means that inferior projects that should otherwise have been dismissed stand a better chance of being approved (Samset, 2010).
One author stated that cost overruns have become so common in the United States that it is no longer a question of systematic underestimation, but that cost deviations have become the norm (Pinto, 2006). He claimed that a culture has developed where decision makers no longer see any reason to give credence to figures presented in the early phase, but acknowledge already at that stage that cost overruns will occur. If this is the case, it is highly unfortunate, and can serve as an appropriate background for justifying measures taken in Norway in recent years aimed to reduce cost overruns in major projects.
In Norway, the problem got particular attention in 1986 when the new headquarters of the National Bank was being planned. Independent experts raised doubt about the official estimate presented to the Parliament. According to their calculations, it would be five times higher. The case became a political issue and hit the newspaper headlines. After the project was completed, it turned out that the external experts were right; the final cost was more than five times the initial estimate. The problem was common and persistent. During the early years of offshore oil and gas exploration and production, there were projects with tremendous cost overruns. In construction of the high speed shuttle train between the capital and the main airport, the final cost was twice as high as the estimate. And in the health sector, several hospitals had large cost increases due to long-lasting implementation periods and with considerable scope changes.
Project Governance in Norway
In 1997, the Norwegian government initiated a study to review the systems for planning, implementation, and monitoring of large public investment projects. The reason for this was both the negative experiences with cost overruns and delays, and that of limited viability of some investments. The study reviewed 11 project cases in the transport, defense, and construction sectors. It was led by a steering committee with participation from the Ministry of Finance and the responsible sectorial ministries, and focused specifically on whether the documentation that provided the basis for decision was adequate when the project was approved, and whether project implementation was satisfactory.
The study (Berg et.al., 1999) found that only three projects were completed within the original budget. The projects had a total budget of about US$1 billion (NOK 5.5 billion). Cost overruns for eight of the projects were as much as 84%. The cost overruns in three of these were particularly high (up to 500%). The study concluded that underlying documentation was deficient in a number of projects, and failures in the initial phase of the project prior to the decision to go ahead were generally the main cause for significant cost overruns during implementation. More specifically, there were
- Unsatisfactory analysis of societal needs, and no analysis of alternative conceptual solutions.
- Presentation of projects to Parliament at a premature level of investigation.
- Inadequate use of cost-benefit analysis, and false assumptions.
- Inadequate assessment of uncertainty associated with cost estimates a number of factors related to procedures, qualifications, responsibilities, etc. that caused problems during the execution of the project.
At about the same, time a government white paper on lessons from investments in North Sea oil exploration was published (NOU 1999:11). This report showed that in a sample of 13 projects all had cost overruns between 17% and 107%, averaging 37% of a total investment of about US$6 billion. For all such projects that were approved in the period 1994-1998, there was an average cost overrun of 13%.
The findings in these two studies suggested implicitly that in the 1990s, cost overruns could be expected to be between 20% and 40% in major public investment projects. A normalization of the situation at this level was not considered acceptable; hence Berg et al. (1999) proposed the introduction of an external quality assurance scheme in the decision phase for the largest public projects.
The scheme was introduced year 2000, with a restricted focus of improving budgetary compliance in public investment projects and avoiding major cost overruns (i.e., to ensure operational project success). This is today known as QA2. From 2005 onwards, the scheme was expanded to include quality assurance of the early choice of conceptual solution as well, that is, to ensure tactical and strategic project success (Qa1). The intention was to make sure that the right projects get started, and to dismiss unviable projects.
In this context, the term ”project governance” refers to the processes, systems, and regulations that the financing party (in this case society, represented by the Ministry of Finance) must have in place to ensure that projects are successful. The Norwegian QA scheme is an example of such an arrangement.
Governance regimes pertaining to major investment projects may be more or less detailed. Previous studies indicate that a good approach is to establish general requirements for structures, processes, results, etc., but not interfere in project implementation as such (Samset , Berg, & Klakegg, 2006). The QA scheme has established requirements for the type of documentation that must exist, but does not require that public agencies use specific tools, formats, etc. The scheme, therefore, does not add considerably to the burden of bureaucracy, since it requires documentation that agencies with good management systems will nevertheless have to prepare for these major public investments. Neither does it interfere during implementation once the project has been initiated. This is in line with the current reform processes that aim towards “freedom with responsibility” or performance management (often referred to as new public management (Christensen, 2009). The idea is that this provides the best pre-conditions for efficiency.
The QA scheme has only two decision points and associated documentation requirements (see Figure 1). This is after the pre-study phase (QA1), and after the pre-project phase (QA2). This is a general overriding model for all major projects in several sectors. QA1 is meant to secure tactical and strategic success, and is designed to assess the outcome and long term achievements of the project. QA2 is meant to ensure the operational success, and is aimed to ensure that cost frames are realistic and that the project outputs are produced on time and in a cost-efficient manner.
Figure 1: The Norwegian QA scheme involves two overarching decision points.
What is added by the QA scheme is the review done by external quality assurers. They are highly qualified consultants preapproved by the Ministry of Finance, to review the documentation before it is presented to political decision makers. The idea is that such an independent review has a disciplining effect and thus an intrinsic value. It is also understood that it could be simplified in cases where supporting documents are of good quality. The quality assurers at QA1 give their recommendation regarding the conceptual solution, strategic framework, and guidance, but the final choice of the conceptual solution is left with political decision makers. Accordingly, the quality assurers at QA2 give their recommendation regarding the cost frame and management of the project, but again the final decision is a political one.
Public Projects Subjected to External Quality Assurance
At present, 13 years after the scheme was introduced, about 160 QA2 reviews have been made. The QA1 scheme has been in operation for eight years and about 60 QA1 reports have been produced.
The projects (with few exceptions) represent major public investments with an expected investment cost above the threshold value of about US$100 million. The cost estimate amounts to US$100 million–US$600 million for most of the projects, while a few have a much higher value. The acquisition of new fighter aircraft, for instance, is estimated to US$9 billion.
About half of the projects fall under the Ministry of Transport’s responsibility (road and rail), then follows the Ministry of Defense (combat aircraft, combat vehicles, weapon systems, etc.). The remainder is construction projects such as public buildings under various ministries.
The first projects that underwent QA2 in 2000, have since undergone an engineering and construction phase that typically takes 5-10 years. As of 2013, approximately 40 projects have been completed and come into the operational phase. For these, it is now possible to observe the final cost, time, quality of delivery, etc.—in other words, indicators for the project’s operational success. By comparing the results with the situation in the 1990s before the QA scheme was introduced, it may also be possible to infer something about the scheme’s impact.
The proposed projects that have been through QA1, however, will first undergo a pre-project phase with subsequent QA2 review, then detailed planning/engineering and then an implementation phase. To date, none of these have been completed. They would also have to be at least three-to-five years into the operational phase before they can be evaluated in a tactical and strategic perspective. The focus in the rest of this paper is, therefore, primarily on the effects of QA2.
Lessons Regarding QA2 (Operational Project Success)
As mentioned earlier, QA2 is conducted at the end of the pre-project phase, before requesting Parliament to approve the project cost frame. The input to the QA2 review is produced by the respective government agencies, which in turn will be responsible for following up the resulting recommendations. The sectorial ministries have the overall responsibility as owners and will commonly be the ones that establish the agencies’ steering frame and control the use of contingency reserves (see Figure 2).
The control aspect is essential in the QA2 review, to ascertain that the basis for the appropriation proposal to the Parliament is sufficient. But it has also a forward-looking perspective to ascertain that key challenges in the implementation of the project are identified. Assuring operational success is thus important: the agreed project output shall be produced in the most time and cost efficient manner. The agency is obliged to provide the following documentation as input to the quality assurance review:
- The overall management document (steering document). This will provide an overview of all key aspects of the project, its objectives and management framework and project strategy.
- A complete base estimate for costs and if relevant also income/revenue
- An assessment of at least two alternative contracting strategies
Figure 2: Stochastic cost estimation.
The quality assurer is assigned to review and verify these documents and make a separate analysis of the investment cost based on stochastic (probability-based) cost estimation. Based on this, the quality assurer gives his or her recommendations regarding:
- Proposed cost frame including necessary contingency reserves, and the agency’s steering frame.
- How the project should be managed in order to keep within the cost frame, including the management and authorization of contingency reserves.
Simple deterministic cost estimates are often systematically skewed and also do not provide sufficient assurance that the cost frame eventually adopted by Parliament will hold. By means of stochastic estimation, either based on mathematical-analytical methods or simulation tools, the result is a cumulative probability distribution of investment cost as shown in Figure 2. The proposed cost frame is normally P85 with deductions for possible simplifications and reductions (reduction list) that can be handled during implementation if the cost frame would be in danger of being exceeded. The agency’s steering frame is lower, normally at the P50-level, in order to avoid incentives to use contingency reserves. The agency must apply to its Ministry to draw on the contingency reserve. (The steering frame available for the project manager may be even lower).
Figure 3: The relationship between the estimates made by the agency and the quality assurers, their recommendations, and the cost frames that are finally adopted for the project.
The Parliament and the responsible ministry are not required to follow these recommendations. The final overall cost frame for the project is decided by Parliament. Then the ministry will determine the steering frame for the executing agency. This is illustrated in Figure 3.
The Final Investment Cost of the First 40 Completed Projects
A research program initiated by the Ministry of Finance has been designed to study the effects of the QA regime, in order to help improve it continuously (see: www.concept.ntnu.no). Researchers have collected data on the first 40 projects that underwent QA2, where the final settlement of contracts were concluded or progressed so far that the final investment cost was known. Of these, about half were road projects (21). The other projects included building construction (7), railway (6) and defense projects (6). This is fairly representative of the sectorial distribution of investment projects under the QA scheme to date.
Figure 4: Deviation between the final cost and the cost frame approved by Parliament (N=40)
Final Cost Relative to Approved Cost Frame
Figure 4 shows the difference between the final cost and the cost frame approved by the Parliament, where the latter largely corresponds with the P85 estimate or barely below this level.
The data demonstrates that 32 of the 40 projects (i.e. 80%), were completed within or below the cost frame. Some (mostly road projects) had significant savings, in total about US$1 billion . Eight projects, however, exceeded their cost frames, totaling US$300 million altogether. About half of this was due to one railway project alone. The total net saving for the projects taken as a portfolio was more than US$500 million, or about 7% of the total investment.
Clearly, this is an exceptionally good result compared to what one could expect based on past experience and findings from a number of studies in other countries, as noted previously.
The number of projects in this study is obviously too small to draw any firm conclusions, but the defense sector stands out, with all projects coming within the cost frame. The road sector represents about the average for the entire sample, with 80% within the cost frame, while the railway and building construction sectors performed somewhat poorer, respectively 67% and 71%.
Figure 5 shows the projects over- and undershoot in absolute terms, sorted according to the size of the project. It demonstrates that all eight projects with cost overrun are smaller projects, (i.e., well below NOK 2 billion). It must be noted that the majority of the projects in the sample are of this size, including those that are within the cost frame. It is interesting to note, however, that all the largest projects were completed with cost savings, deviating positively from the cost frame.
Figure 5: Difference between the final cost and the approved cost frame, by the size of projects. Only the smaller projects had cost overruns.
Another factor that may influence the probability of cost overrun is the date of the commissioning of the project. One could imagine that the outcome is affected positively or negatively by learning effects, or effects of economic cycles that have not been captured by the cost analysis.
Figure 6 shows the difference between the final cost and the approved cost frame for the projects, now sorted by time of inception, from 1999 to 2008. We then find (with one extreme exception) a tendency that cost overruns have been in the latter part of the period. This may be due to strong increases in costs in the construction industry that occurred towards the end of this period. It could also be that the subsequent global financial crisis of 2007–2008 have had unforeseen consequences. Beyond this, it is difficult to determine any cause of the vague tendency that is observed.
Figure 6: Deviations between final cost and approved cost frame, by the time of commissioning of the project.
Estimates and Recommended Cost Frames (QA2)
The QA2 review results in recommendations regarding cost and steering frames, and how the project should be managed in order to keep within these frames. The recommendations are only advisory, so a relevant question is to ask to what extent the recommendations are followed when the cost frame is determined.
Final Cost in Relation to the Agency’s Steering Frame
The steering frame for the executing agency coincides, as mentioned, to a large extent with the estimated P50 value. Ideally, all projects should be completed at the expected value. However, given the uncertainty associated with implementation, one must not only expect, but also accept deviations (in both directions). The principle is that if a portfolio of several projects together is completed with equal sizes of overruns and underruns in relation to the steering frame, the average for the whole portfolio will still be around the expected value.
Figure 7: Deviation between final cost and the agreed steering frame for the project. N=40
The differences between the final cost and the steering frames are illustrated in Figure 7, which shows that this is exactly the case in the projects in this study. The differences are symmetrically distributed about the expected value, and the portfolio as a whole will correspond with the P50 value fairly accurately. Overall, the portfolio of projects has been executed with a total cost overrun only about US$160 million in absolute value.
This is a sign of good cost control at the portfolio level. It is also a sign that the methodology of stochastic cost estimation is well suited for the purpose.
The Cost Frame
Data shows that in about 70% of the cases, the cost frames approved by Parliament are identical with the quality assurers’ recommendations. In other projects, there are minor deviations (up to +/- 6%), mainly adjusted upward.
The quality assurer’s recommendation is, in turn, identical to the P85 estimate for approximately one third of the cases; in a few cases, it is higher (e.g., P90), while in more than half of the cases, the proposed cost frame is lower, since the reduction list has been deducted (see Figures 2 and 3).
The Steering Frame
If we instead look at the steering frame, we find that in 54% of the cases, the final steering frame is the same as recommended by the quality assurers. In the remaining projects, with two exceptions, the deviations are within +/- 10%, and the tendency is to reduce the steering frame.
The recommended steering frame is in turn identical or very close to the P50 estimate in two thirds of the cases. In the remaining projects the deviations are small and within the P45 - P55 range.
Evaluating Operational Success in 23 Projects
In addition to studying the cost figures in 40 projects, we looked in more detail at the implementation of the 23 first completed projects. Large amounts of data were collected regarding how these 23 projects were implemented, with the aim to assess both the outputs and processes, and how the projects were organized and managed. The main conclusion was that project outputs by and large were achieved as planned both in terms of time, cost, and quality, and that the projects were essentially well organized and executed. The implementation phase lasted five years on average. Only two of the projects lagged more than three months behind schedule.
Researchers also looked into the extent of changes that had taken place during the implementation, with a view to identify any quality improvements or scope increases beyond what was planned. They concluded that there were no cases where cost overrun could be explained by scope changes. It was not always well documented what was the cause when the steering frames were adjusted during the implementation period (or contingency reserves were used), and clear criteria for such amendments were not always established.
Organization and management of the projects were considered favorably, progression was good and actions were taken as needed. The weaker points were uncertainty management and contract management in some of the earlier projects, but uncertainty management in these cases was probably not representative for later projects where this has been highlighted, and both methodology and procedures have improved. It was suggested that project governance at the ministry level could have been better, but the researchers did not go deeply into these issues.
Lessons Regarding QA1 (Tactical AND Strategic Project Success)
Nearly eight years after the QA1 scheme was introduced, it is still too early to evaluate its effects, since none of the projects that have been reviewed so far have yet been completed and put into operation. Below is nevertheless an account of what we know so far about the projects that have undergone QA1 and, indirectly, the attitudinal changes and the spin-off effects in government, industry, and academia after the QA1 scheme was introduced.
What the Scheme Involves
QA1 is implemented at the end of the pre-study stage, before a decision by the Cabinet whether or not to proceed with the pre-project phase. The ultimate aim is that the chosen concept is the one that is considered the best use of public funds. The choice of concept is a political decision to be made by the Cabinet, while the quality assurer’s role is restricted to assessing the quality of the documents supporting the decision.
The responsible ministry and/or executing agency is required to prepare a Conceptual Appraisal (CA), which should include the following chapters:
- A needs analysis mapping all stakeholders and assessing the project’s relevance in relation to societal needs and priorities.
- Overall strategy defining the project’s goal and purpose (first order and long-term effects).
- Overall requirements specifying other essential requirements.
- Possibilities study. Needs, goals, purpose, and requirements will together constitute an “opportunity space” and allow alternative conceptual solutions to be identified.
- Alternatives analysis, which should include the zero option and at least two alternative conceptual solutions. The alternatives should be subjected to a benefit-cost analysis.
- Guidelines for the pre-project phase for the recommended concept.
The quality assurer performs an independent review of the conceptual appraisal and also conducts his or her own uncertainty analysis and economic analysis, as well as provides recommendations regarding the decision-making strategy. Alternatives should be ranked based on an assessment of monetary and non-monetary costs and benefits. Quality assurance thus involves a separate independent professional review, in addition to providing a control of assessments and estimates done by the executing agencies, which might have a vested interest to overestimate benefits and underestimate costs of certain preferred alternative solutions.
The Experiences with Conceptual Appraisals and QA1 After Eight Years
In the period from 2006 to March 2013, 57 investment projects with full or partial government funding were subjected to QA1. Samset, Andersen, and Austeng (2013) did a review of a sample of 17 cases and concluded that the quality of both conceptual appraisal reports and quality assurance reports had improved steadily over time and that there was a convergence toward a common best practice. Exchange of experience through a wide range of quality forums organized by the Ministry of Finance had resulted in a number of guidelines, including a guide on conceptual appraisal (Ministry of Finance, 2010b). Several government agencies had also developed their own guidelines and templates for conceptual appraisal work (see e.g. NPRA and Jernbaneverket [2006], Norwegian Public Roads Authority [2010], Norwegian Rail Authority [2011], and the Ministry of Defense [undated]). The military has integrated the conceptual appraisal into their project model PRINSIX. Also, the requirement of conceptual appraisals has provided an incentive to keep guides and templates for economic analysis up to date in ministries and agencies.
There are reviews regarding how the conceptual appraisal/QA1 process perform in the transport sector (see, e.g., Rasmussen et al. [2010], Norwegian Public Roads Authority [2012], and Bjertnæs [2012]). Some of the findings are that the conceptual appraisal/QA1 process can be time- and resource-consuming, but the main picture is that agencies seem to benefit from the scheme. Conceptual appraisals in particular represent a more systematic approach to early identification of project ideas than in the past. Planners are forced to take a broader perspective and discuss societal aspects, rather than going straight to the issue of selecting road sections and their technical solutions. This allows for ideas to mature and stimulates creativity in the agencies. It also increases the likelihood that the best option will be included in the analysis.
The QA scheme provides the ministry as well as the government with more direct influence in the early stages of a process than before, and in relation to local stakeholders who traditionally have had significant influence, especially in road projects. The ministry is drawn more actively into the process and can provide guidance on what kind of needs and societal objectives are essential, before specific solutions are discussed in consultation with local stakeholders.
The above-mentioned studies also indicate that there is still room for improvement when it comes to practice and the content of the analysis. A recurrent problem is that the selection of a conceptual solution often is done before the conceptual appraisal, either as the result of perverse incentives at local level, path dependency in the agencies, or as the result of political constraints and limitations. There are indications, however, of improvements.
Both practical and fundamental issues related to economic analysis have been highlighted by recent studies. In 2011, an expert committee was appointed by Royal Decree to review the cost-benefit analysis framework. They recommended more equal and scientifically founded practice concerning aspects such as the valuation of life, the social cost of carbon, real price adjustment, the discount rate, the analysis period and the calculation of residual values (NOU 2012:16). Based on this report, the Ministry of Finance is now updating the guidelines on cost-benefit analysis. One review of current practices based on the first 24 QA1 reports was that they often had a relatively short and narrow economic perspective (Lædre, Volden, & Haavaldsen, 2012). Another study, (Norwegian Public Roads Authority, 2012), suggests that the quality assurers seem to give disproportionate attention to economic and financial considerations, and that there is a need for a more balanced presentation of economic impacts and achievement of various objectives.
Experience so far indicates that QA1 has helped improve the basis for decisions and is taken seriously by policymakers. Recommendations are followed in two-thirds of the cases (Concept Newsletter, 2011-4).
This can probably be explained by the fact that political authorities are now being presented with the case at an earlier stage than before, when the pressure to choose between specific solutions has not yet fully materialized. There is evidence to suggest that an independent QA report documenting that an investment is poorly justified may give weight to decision makers and be essential for the government to make a sound decision in a controversial case.
The impact of the QA1 scheme can also be less visible, for instance in cases where the opportunity space is broadened during the process as the result of advice by the quality assurers. Their role is not only as controller but also as adviser. In several cases, new and improved versions of the conceptual appraisal document have been produced during the QA process. Also, there is reason to believe—but difficult to prove—that many of the least-thought-out investment proposals now are screened out before they even reach the conceptual appraisal/QA1 stage. This would be the result of improved processes and procedures in the involved ministries and agencies, as well as the deterrence effect itself, and could probably be the most important beneficial effect of the QA scheme.
There are also other spin-offs to be observed. Other agencies not subject to the QA scheme have voluntarily introduced variants of the scheme. This includes investment projects run by health authorities (Myrbostad, Rohde, Martinussen, & M. Lauvsnes, 2010), for high voltage electricity transmission and distribution projects, and investment projects by the Oslo municipal authority (Oslo Kommune, 2011). Other countries have shown interest in the Norwegian scheme. In Sweden, a variant of the conceptual appraisal report was introduced in 2013 as a new step in early planning, directly inspired by the Norwegian scheme. The Province of Quebec, Canada has introduced a similar scheme, and the U.K. has recently established a Major Projects Authority, directly under the Cabinet Office, which seems to operate on similar principles.
Conclusions
The purpose of QA2 is to improve cost management and ensure operational success more generally. The situation in the 1990s was that major cost overruns were a norm rather than an exception, both in Norway and other countries. International research (Flybjerg et al., 2003) has shown that the situation has neither improved nor worsened over the past 70 years.
The results from the first 40 projects that have been subjected to QA2 show that about 80% have been completed within the cost frame. This is a remarkable improvement compared to the past. The difference between the final cost and the steering frame is almost symmetrical around the expected value. This means that at the portfolio level, the state is now in good control of the cost in major investment projects. The fact that the deviations are both positive and negative to the same extent also suggests that there is no incentive to spend contingency reserves.
The cost frames are largely based on the recommendations of the QA2 reports and based on stochastic cost estimation. This indicates that the quantitative uncertainty analysis provides robust results for a sufficiently large portfolio of projects. Most risk factors that do in deed materialize are identified in the QA reports. The practice of establishing a lower steering frame for the executing agency (typically at the P50 level) has probably been an important step to provide incentives for cost efficiency.
What we have not discussed in this paper is the question of how the project content, scope and cost estimate evolves in the process that precedes QA2. To avoid cost overruns in relation to the cost frame is no doubt important. But before the cost frame is set, projects have commonly been through a long front-end phase with great expectations among user groups and stakeholders. When the project is presented to Parliament, it may be too late to turn down the proposal, even in cases where the QA2 estimate is higher than would normally be accepted. It is at the QA1 stage that the cost estimate is compared with expected benefits to decide whether the project is worthwhile to implement. If the cost estimate, however, increases between QA1 and QA2, the assessment of the project’s tactical and strategic success from the QA1 stage will no longer be valid. Only now can we observe the first projects that have been subjected to both QA1 and QA2, and the researchers will expand their focus correspondingly.
The choice of concept is a key issue and directly related to the project’s tactical and strategic success. Nearly eight years after the first QA1 report, what can be said about the effects of the scheme is still very limited, since none of the QA1 projects have yet been completed.
It seems clear, however, that systematic appraisal of the choice of conceptual solutions has benefits. We have observed that the quality assurer’s recommended choice of concept is largely taken into account by decision makers. We are also starting to see some indirect spinoffs of the scheme, in terms of other ministries or organizations establishing similar systems on a voluntary basis.
The final test whether the QA1 scheme works or not is of course that the projects prove to be both tactically and strategically successful. In the years to come, the Concept Research Program will follow up with systematic evaluation of QA projects to find the answer to this.
The QA scheme in its current configuration appears to be suitable for the purpose for which it was designed. However, governance regimes should not be static. They need to be flexible so that they can be altered if they do not work as intended or if changes in operating conditions and characteristics of the projects should necessitate change.
