Success vs. failure
what is the difference between the best and worst projects?
What determines the success or failure of a project? What practices lead to consistently better or more competitive results, and what factors cause capital projects to fail? This paper will discuss the results of an extensive research study carried out on industrial projects in the last few years. The findings of this study reveal the best course for project professionals to achieve project success.
What are the most frequent causes of failure in capital projects? Further, what factors lead to project success? Understanding the drivers of success and failure for capital projects is valuable because it will enable project teams to avoid costly errors. In addition, by gaining an understanding of these factors, business can enforce the use of practices known to drive successful projects, and increase the likelihood of obtaining competitive and lasting results.
The research study (Bransfield & Bhat, 2008) was carried out by Independent Project Analysis, Inc. (IPA) on projects that were authorized after year 2000 on the key differences between successful projects and those that fail to reach their objectives. The research not only confirmed what is already known about some practices in the industry, it also refuted some common myths about other practices. To illustrate the effects that these practices have on projects, four case studies are discussed; these actual projects were selected because they clearly illustrate the lessons learned from this research.
Definition of “Best” and “Worst” Projects
The dataset for this study comprised of projects that had excellent results and those with bad results. These 73 process industry projects had an average authorization date of January 1, 2000, and an average completion date before December 31, 2007. The projects were executed by 49 different organizations. The total cost of each project ranged between US$10 million and US$500 million. These projects were selected and classified as either a “Best” or “Worst” project; the criteria for these categories are defined below.
The Best Projects
A project was considered to be in the “Best” category if it met all of the following criteria:
- A total cost that was lower than the industry average for similar projects by 10 percent or more
- An execution duration or overall cycle time that was industry average or faster
- Safety performance that included no fatalities, and
- No serious operational problems during startup or in the first 12 months of operation
The Worst Projects
A project was considered to in the “Worst” category if it met all of the following criteria:
- A total cost that was higher than the industry average for similar projects by 20 percent or more, and
- An execution or cycle time duration that was slower than industry average by 20 percent or more
IPA also investigated specific project features to determine if they had an effect on a project's success or failure, including project location (country or region), contracting strategy (e.g., lump-sum, reimbursable), project type (e.g., greenfield, colocated, expansion), industrial sector, and use of new process technology. However, none of these features were found to have a major influence on project results.
Having defined both groups, it was observed that, compared with the industry average:
- The Best Projects were, on average, 18 percent lower in cost, 8 percent faster in cycle time (total length of the project) and 10 percent faster in execution.
- The Worst Projects were, on average, 42 percent higher in cost, 49 percent slower in cycle time, and 29 percent slower in execution.
In the next section, four case studies are presented that discuss the level of planning that was completed prior to approval, and how these planning practices affected project outcomes.
We evaluate the level of project definition just prior to execution because this is a common point in most companies' project management processes at which owner companies make full-funding approval decisions. The best project management processes have review gates at select points in project planning to assess the level of maturity of project definition, and either cancel the project or approve it to proceed to the next phase. According to the PMBOK® Guide (PMI, 2008), the completion of each phase represents a natural point for the reassessment of efforts and modification of the time period of the project, if necessary.
First Case Study: Bad Planning → Bad Results
Company A conducted an expansion program mandated by new environmental regulations. The project team felt that the project was constrained by an aggressive regulatory deadline. We focus on one project in the program that was estimated to cost US$132 million.
Although the project was authorized, it was not ready to enter execution: the plot plans were not finalized, the soil survey (test drillings, contamination, and groundwater level) of the site was not done, none of the operating risk analyses (HAZOP study) had been completed, the specifications of the main equipment were not defined, and the project schedule included only summary-level milestones. Although some plans were missing, business justified its approval because it felt that the deadline was too urgent to allow time for optimal planning.
During execution, the project encountered issues that negatively affected the project's cost and schedule performance, many of which could have been avoided if Best Practices had been followed in the planning phase. After full-funds authorization, the entire project team was replaced because of company standard practice to use two separate organizations for developing and executing projects. Management decided to shortcut planning and enter execution with incomplete basic engineering. In addition, the project was authorized without defining the necessary capacity for the new unit, which is fundamental for making decisions regarding equipment specifications and sizing. As the project team was trying to work around this issue, top management decided to acquire the equipment on behalf of the project team in an attempt to reduce the purchasing costs, which essentially negated the ongoing negotiations with vendors carried out by the team. This resulted in delays to procurement, which, in turn, delayed the project's execution.
Finally, when the regulatory agency delayed the compliance date for meeting the new regulation, Company A management saw an “opportunity” to improve the economics of the project by adding to the project scope. The numerous delays caused financial strain on the engineering contractor, which contributed to its eventual bankruptcy during execution, causing further delays and team member turnovers. Here is what happened with this project:
- During the first 6 months of execution, the project was ahead of its milestone schedule due to the overlapping of phases (“fast tracking”). A performance bonus was even paid to the construction contractor.
- Various changes had to be made to the design due to the inconsistencies between basic engineering and detailed engineering; specifications for materials and equipment were affected which, in turn, affected construction.
- The cost of the project was 24 percent lower than planned because the initial budget was insufficiently detailed and included excessive contingency. Some company cultures tolerate underruns; however, underruns signal a cost-inefficient system: the additional funds that were tied up in the project budget could have been used elsewhere.
- The schedule overran by more than 16 percent.
- During startup there was a fatality, which led to a complete plant shutdown that lasted 45 days.
This case study shows the direct consequences of authorizing projects without the necessary level of definition and failure to follow the process of freezing scope.
Second Case Study: Good Planning → Bad Results
Company B had to carry out a revamp project of a unit for environmental reasons. This US$150 million project had a deadline imposed by a new regulation. The schedule became aggressive during execution due to the business decision to delay the project start.
The project team followed many Best Practices during planning: all plot plans were completed and approved by operations personnel, the facilities had been laser scanned to identify any interferences, all soil surveys and risk analyses had been completed, and all procedures for environmental licenses were being carried out by the project team. In addition, all of the necessary team representatives were formally appointed (operations, maintenance, safety, and construction) and participated in the planning stage; a detailed, resource-loaded schedule had been developed and all basic engineering studies were completed and approved before authorization.
Thus, this project had all of the necessary “ingredients” to be successful. However, business leaders decided to delay the authorization of the project, resulting in compression of the execution schedule. After the project was authorized, it became apparent that the team had not done enough to understand local union labor practices. Although the team had discussed labor requirements and availability with the union shop prior to authorization, the project team did not understand union practices such as on-the-job training requirements and fire watch requirements, which caused labor costs to grow.
The team had also planned to use a modular construction strategy, but some decisions on modularization were still open while the project moved into execution. Because the technology to be used was not completely known by the company, the team decided to carry out some tests with the suppliers before purchasing the main equipment. However, due to problems caused by the suppliers, the acquisition phase was delayed 11 months. Also, during execution, top management saw several “opportunities” to add to the project scope, for an additional US$50 million. Here are the results of the project:
- Minor design changes added more than US$15 million during execution.
- There was a 4-month delay to detailed engineering (although construction was only delayed by 1 month because the team overlapped the phases more).
- A change in sequence of field work was meant to keep workers busy while engineering was being completed.
- Overtime was used for 25 percent of the total labor time to meet the deadline (added more than US$11 million in costs).
- Labor cost doubled due to the overtime and the lack of knowledge of the local labor situation.
- Total project cost was more than 30 percent higher, and a cycle time more than 15 percent longer, than industry average for similar projects.
- There were more frequent shutdowns due to the lower than expected lifespan of the catalyst, forcing the company to carry out more frequent maintenance shutdowns than planned.
These results show that having quality planning does not guarantee good results if changes are made during execution and the team does not have real control of the project. Moreover, allowing business to use “opportunities” to make scope changes erodes the efforts of the team in definition.
Third Case Study: Average Planning → Bad Results
Company C had to execute a simple reactor replacement project totaling US$17 million. Its primary objective was to improve the reliability of the unit.
This project's planning effort was of moderate quality. The existing facilities were studied, the location of the cranes for the replacement process was defined, and the lifting plan was finished. However, there were some gaps from Best Practice: a shutdown would be carried out to change the reactor that involved many different projects that were not sufficiently planned or integrated; operations and maintenance personnel had not yet validated basic engineering; and some known interfaces with the existing facilities had not been detailed.
When the project entered execution, management assigned the project team to an unrelated capital project on an adjacent unit, and asked a contractor to assume responsibility for completing the reactor replacement project. The current project team was asked to provide oversight of the contractor. This was done because top management wanted to “test” the performance of this contractor; the original project team did not agree with this decision.
The project manager contractor faced many difficulties in staffing the project team because the project was located far from its headquarters. The contracted company's project manager was replaced twice and no formal change register was kept during execution. Even after the project cost had grown, the contracted company started to demand incentives set forth in the contract because the performance indicators of the project were never established. Top management paid these incentives. The original project team did not agree with this decision and refused to assume responsibility for the project because it wanted top management to see the effects of its mistakes. Because of these issues, the project had these results:
- Significant increase in construction labor hours due to congestion caused by the two projects being executed concurrently in close proximity (result of insufficient planning for the interfaces).
- 40 percent increase in the total cost. Overall, the project was 71 percent more expensive than industry average for other similar projects.
- 65 percent longer than industry average construction duration.
- The owner's project team did not want to see its names associated with this project because it understood that the failure had been caused by top management's decisions.
With average planning, the lack of an owner team to lead and supervise the work, and project management left to a contracted company that did not have the necessary resources, the project suffered schedule deviations and cost growth, and was not competitive.
Fourth Case Study: Average Planning → Good Results
Company D started a US$50 million project to increase the production capacity of polystyrene by more than 40 percent to take advantage of a market window.
At authorization, the project had not done an operation risk analysis. A detailed, resource-loaded schedule had not been developed, and the cost estimate had not been validated. However, all basic engineering had been developed, reviewed, and approved; top management support had been formalized in the project plan; the team included all necessary functions, and roles and responsibilities had been developed.
As a result, the project had:
- No changes during the entire life cycle
- 15 percent lower than industry average overall cost
- An industry average cycle time driven by a slow planning phase offset by a fast execution phase
- 15 percent better than planned operability for the new production line
How were these results possible with average planning? The team compensated for the gaps in planning with excellent discipline during execution. The project team members and the contracted parties were well aligned during execution, and top management was committed to the project. Moreover, during execution a detailed schedule was developed and physical document and material progressing was used.
Clear definition of objectives, the support of top management, and the combination of an integrated team and discipline during execution resulted in very competitive results.
Main Results of the Research: Identification of the Practices That Lead to Success
The objective of this research was to identify the behavior and practices of each project to understand how practices in the planning and execution phases affected project outcomes. The research concluded that projects that obtained the best outcomes were those that had the highest level of definition before authorization. As would be expected, the projects with the worst results were those that were missing important planning components at authorization, such as a detailed schedule, complete basic engineering approved by operations and maintenance, a risk analysis with mitigation plans, complete responsibility matrix, etc.
Exhibit 1 shows the comparison of the planning level as measured by the Front-End Loading (FEL) Index of the Best and Worst projects. A Best Practical rating implies that the team completed all necessary deliverables for a good project planning before authorization; that is, the project was fully prepared to enter execution. The FEL Index runs from Best Practical (the best level of planning) to Screening (the worst level of planning).
Exhibit 1: Comparison of Planning Levels
It was also verified that the level of team development was better for the Best Projects, as shown above. Team development ratings are based on alignment of the parties involved in relation to the project objectives, clear definition of the roles and responsibilities of each team member, active participation of all relevant members of the team (project manager, operations and maintenance representatives, construction manager, etc.), main risks being identified, and the use of a standard company project implementation process.
To increase the chances of good results, execution discipline is essential. Project control and the continuity of the core project team members are crucial. If the team cannot keep the project from deviating from plan, the benefits of good definition work will likely be lost. Further, turnover of team members during development and/or execution can cause delays, continuity problems, and changes.
Based on this research, the Best Projects had an average Project Control Index (PCI) of Good and the Worst Projects had an average PCI of Deficient. A Good PCI rating means that the project had its cost estimate validated by an internal group specialized in cost estimating, it kept regular and detailed measures of its physical progress with detailed reports issued every 2 weeks (or more frequently), and a cost control specialist was appointed to accompany the project.
The Best Projects also had a lower rate of project manager turnover. Just 17 percent of the Best Projects had project manager turnover, while the rate for the Worst Projects was 58 percent, as shown in Exhibit 2.
Exhibit 2: Rates for Worst and Best Projects
Importance of Predictability and Competitiveness for Successful Results
A project's success is often measured in terms of its cost and schedule competitiveness and predictability. A predictable project is one that was completed on schedule and within budget. A competitive project is one that was completed with a schedule that was faster and a cost that was lower than the average of other similar projects in industry. As can be expected, a project can be predictable, but not necessarily competitive, and vice versa.
IPA research shows that setting competitive cost and schedule targets is important for project success. Projects that are well defined and that set competitive cost and schedule targets tend to achieve competitive cost and schedule results. However, projects with industry average targets tend to achieve industry average results. The research found that many of the Worst Projects set uncompetitive targets that they would often overrun, suggesting that setting conservative targets does not safeguard a project from disaster. Exhibit 3 shows the comparison between the estimated and actual cost and schedule results of the assessed projects.
Exhibit 3: Comparison of Estimated and Acutal Cost and Schedule Results
One might also assume that because the Best Projects have better planning, they must take longer in the project definition which, in turn, would lead to less competitive schedule results. However, research (Bransfield & Bhat, 2008) has proven this to be false. In fact, the projects in this sample that took longer in planning had the worst results. On average, the Worst Projects had an FEL phase that was roughly 71 percent longer than the average for the Best Projects. Thus, the time spent in FEL does not determine the quality of planning; rather, it is the deliverables completed in FEL that are most correlated with results.
Lessons Learned From the Best and (Primarily) the Worst Projects
Lessons learned from this research can be used by both project professionals and business sponsors to improve their future project results.
- The fundamentals are important―The level of definition (Front-End Loading) of the project before authorization is strongly correlated with good outcomes. A well-developed owner team is also important.
- Bad practices during execution will erode the benefits of good planning―If a project is well defined at authorization but experiences major design and/or scope changes, the benefits of definition are eroded. Thus, maintaining good discipline during execution is fundamental for good results.
- Uncompetitive targets lead to uncompetitive results―Research has shown that setting competitive targets leads to competitive results, while industry average or uncompetitive targets tend to lead to industry average or uncompetitive results. A project may have good plans in place, but if its targets are not competitive, it may have poor results. However, if a project has been poorly planned, it can be dangerous to set aggressive targets because the project has not been positioned to be competitive.
- Business can ensure consistency if a work process is followed―If business does not succeed in keeping discipline during execution, or if it interferes in the process by imposing late changes, projects are likely to fail in the effort to be competitive.
Bransfield, S. & Bhat, S. (March 2008). Best and Worst. IPA, Industry Benchmarking Consortium 2008, Herndon, Virginia, United States.
Project Management Institute. (2008) A Guide to the Project Management Body of Knowledge (PMBOK®) (4th. ed.). Newtown Square, PA: Project Management Institute.
© André Augusto Choma, PMP & Swati Bhat, IPA
Originally published as part of Proceedings PMI Global Congress 2010 – Washington D.C.