Revisiting the definition of IT project success
By Yves Cavarec
Project Management and Strategy Consultant
The triple constraint used to be a convenient approach to a successful project. It is still common to use scope, time, and costs in statistics to separate the good from the bad. The Standish Group, for example, continues to uses these traditional criteria to compile its famous Chaos Report (1994). It is high time to use business criteria to redefine the success of projects. And it is high time to take organization impacts into account before considering if a so-called “IT project” is successful or not. The end of this paper gives you an indication of how to settle the right success criteria for your projects.
They say projects fail. Ask the Standish Group and read its publication, the Chaos Report. Since 1994, the Chaos Report has stated that the project management of application software development is in trouble, with a shocking 16% success rate. The Standish Group was not the first, and not the only one, to consider that projects fail, especially IT projects.
Many have tried to understand why and what to do about this big issue. Many researchers and practitioners inquire about the reason for project failure. Morris and Hough (1987), for example, studied 1,653 projects and concluded that typical sources of difficulty were unclear objectives, changing sponsor strategy, poor project definition, technological difficulties, concurrency, inappropriate contracting strategy, unsupportive political environment, lack of top management support, funding difficulties, inadequate manpower, and geophysical conditions.
Before going any further with the cause, are you sure projects fail that much? To make sure, let's understand and stress the success criteria used. Over 90% of studies, including studies like the Chaos Report, and scientific publications mentioning project success, consider success and failure on the triple constraint basis. In their criteria, to be successful, projects must be on time, in budget, and in specifications. Because project success relies on estimates, let's hope the estimates are unbiased!
The first part of this paper will show that this is not the case, unfortunately. Then, we will see that the triple constraint is not the right criteria for evaluating the project success. The third part of the paper introduces a model to identify the right success criteria for your project.
Success Depends on Project Planning
Assuming that the “iron triangle,” also called the “triple constraint,” is the success criteria, success would mean that projects are on time, in budget, and in specifications. Failure indicates a gap between the estimates at the early stage of the project and what is eventually realized. Conventionally, this gap is explained by the context of the project; for example, lack of experience of project planners might explain the project failure. Academic researchers, such as SwedeBent Flyvbjerg, have focused on project estimates activities. They have identified new explanations and unconventional reasons why a project goes over time, over budget, and out of specifications. They found two unconventional causes of project failure: the psychological explanation and the political explanation.
Exhibit 1 – The iron triangle, also called the triple constraint
Conventional Explanations to Project Failure
Lack of experience
Project management requires experience. It is an activity that is usually learned by doing, even though there is a body of knowledge that can be taught — thanks to A Guide to the Project Management Body of Knowledge (PMBOK® Guide). Because of the lack of experience, a project manager could have difficulty keeping his or her project on time, in budget, and in specifications.
Underestimation of time, budget, or risks is often called optimism. Optimism is a cause of underperformance if the project planner is unrealistically confident. Such a forecaster expects an activity to be executed within a low duration or with a low level of resources, when reality shows that the level of resources required for the task is higher.
Black swan and bad luck
In the definition given by Nassim Nicolas Taleb (2007, p 16), a black swan is not a bird but an event difficult to predict based on the given information and with serious consequences. A black swan is about small probabilities. Rare events are merely impossible to predict, so that if they happen on a project, it will necessarily impact the result of the project in terms of time, budget, or scope. The project team is a victim of something unpredictable and should not be blamed for that.
The problem with conventional explanations
Conventional explanations to project failures can generally be decreased through risks management. Experience from previous projects helps in predicting the risks. Quality of forecasts comes from improving data quality and comparing historical information across projects. Most of the events considered as black swans are actually predictable: they do not pass statistical tests.
Does it mean that project planning should be unbiased? Unfortunately, it doesn't. Two other explanations must be considered and they are unconventional: the psychological explanation and the political explanation.
The “planning fallacy” is a tendency for people and organizations to underestimate the duration and the resources needed to complete a task, even when they have experience with similar tasks over-running. The idea was proposed in 1979 by Kahneman and Tversky and demonstrated fifteen years later by Buehler, Griffin, and Ross (1994). Managers make decisions, underestimate costs and time, and overestimate the benefits. More research has documented the planning fallacy in projects (Flyvbjerg, Holm, & Buhl, 2004), in business (Malmendier & Tate, 2003), and in start-up ventures (Dune, Roberts, & Samuelson, 1988).
“Anchoring” on plans is another bias of judgment. The “anchor” is the first version of the estimates. Even when the project planner knows that the anchor is too high or too low, his or her adjustments to it away from it are almost always insufficient. The initial plan was supposed to be realistic and built according to the “EGAP principal” (everything goes according to the plan).
Inside view biases
Planning fallacy and anchoring generally result from an inside view of the project: people who do the estimates should be involved in doing the work. Adopting an outside view will reduce the bias by ignoring the details of the project and comparing them with similar projects. For example, executives adopt an outside view by estimating the gap (i.e., the difference between the forecast and the realized) by applying a proportion of costs or time given by the plan (inside view). The estimate executive formula looks like the following:
Outside view estimate = inside view estimate + % of inside view estimate
Note that the inside view biases were observed decade after decade on planning activities. But estimating is usually a collective activity, involving experts in their profession and not (or not only) project managers. So, it seems that learning by doing is not enough to balance the bias in estimates.
Niccolò Machiavelli writes that “contemporary experience shows that princes who have achieved great things have been those who have given their word lightly, who have known how to trick men with their cunning, and who, in the end, have overcome those abiding by honest principles. (1513, p 99)”
Lying pays off
The second unconventional explanation to a project not being on time, in budget, and in specifications is political. In the model set by Wachs (1986) and adapted by Flyvbjerg, managers overestimate benefits and underestimate costs and time in order to get funding. They purposely amplify the expectation of success and deliberately hide the risks. Due to political pressure, “lying pays off, or at least agents believe it does” but it can be moderated by measures like encouraging transparency and accountability or aligning incentives, Flyvbjerg says.
The survival of the unfittest
If managers intentionally underestimate costs and time and overestimate benefits, it is hard to imagine them telling this to researchers. Nevertheless, in 1990, Wachs in the United States and Flyvbjerg and Cowi in 2004 in the United Kingdom, successfully got managers to talk about using lies to get a project funded. Flyvbjerg (2011) considers that securing approval and funding for projects leads to what he calls an “inverted Darwinism,” because organizations should not select the best projects but the projects that have been made to look the best on paper. The formula to the “survival of the unfittest” would be:
Underestimated costs + overestimated benefits = project approval
Forgetting About the Triple Constraint
There are other reasons why the triple constraint cannot be the measure of project success. First, the increasing complexity and uncertainty make it so we must be prepared to manage changes to the baseline in each project, and the PMBOK® Guide can help us do that. Second, success depends on the point of view. Third, concerning IT projects, the success of a project wouldn't mean the success of the organization that uses it.
You don't know what your project will be
Since World War II and up to the early 1970s, the economic environment was quite stable in western countries. Economic forecasts were reliable at the national level as well as the market level. It was thesame for a project forecast, they were reliable because they were not being altered by a changing environment. Since the 1970s, mass production has gradually overloaded the traditional markets, which have reached their saturation points. Competition progressively became tougher at the local and global levels, and the world has become more and more unstable. Uncertainty is higher today than 40 years ago. Turbulence has increased with rapid change, deregulation, globalization, and innovation. The environment is also more complex today than during the 1970s. It has created a higher interdependence of tasks within an organization, and the world is still growing more complex and less stable, Michel Thiry says (2011).
Changes to the project environment
Changes occur in the project environment. They force the project manager and the project team to reconsider the baseline of the project, in terms of scope, time, and costs. Of course, the project realized is different from the initial plan because it had to adapt to the environment. It is not a symptom of bad health, but proof that the project team can adapt to changes.
The paradox of the project manager
Because of complexity and uncertainty, it is difficult to make the right decision at the right moment. At the beginning of the project, the project manager and the project team know little about the environment. They have the choice and could make any possible decision, but they ignore the consequences of their decision. Decision after decision, the project team understands the project environment better; at the same time, the number of possible decisions is decreasing. At the end of the project, the project team completely understands the consequences of the decisions made. Maybe they think they could have made other choices, but now it's too late! This is the paradox of the project managers: when they can, they don't know and when they know, it's too late.
Exhibit 2 –The paradox of the project manager
Success depends on the stakeholder's point of view
The Channel Tunnel
The Channel Tunnel was a privately funded billion-pound project privately funded. Here is its story, briefly told:
- The Channel Tunnel was constructed between 1987 and 1994.
- The total investment costs at 1985 prices were £2.60 billion. At the 1994 completion, the total construction cost was £4.65 billion (equivalent to £11 billion today), an 80% cost overrun. The cost overrun was partly due to enhanced safety, security, and environmental demands. Financing costs were 140% higher than forecasted.
- Both the freight and passenger traffic forecasts that led to the construction of the tunnel were largely and universally overestimated. A Eurotunnel advertisement, on 7 November 1987 forecasted 30 million passengers. The high score has been 18 million so far. For freight transported on through freight trains, the first year the freight prediction was 7.2 million gross tons; however, the actual 1995 figure was only 1.3 million gross tons.
- Shares in Eurotunnel were issued at £3.50 per share on 9 December 1987. By mid-1989, the price had risen to £11.00. Delays and cost overruns led to the share price dropping to £0.50 in March 2003.
- Ten employees, eight of them British, were killed during construction, between 1987 and 1993.
From what we know now, depending on which stakeholder you are, you might think this project should have been avoided.
- From the point of view of the employees who risked their lives during construction, this project should have been cancelled.
- The shareholders who their savings probably regret having taken part in the project.
- But, from the point of view of the people who travel from Paris or Brussels to London on the Eurostar train, the result is wonderful. It takes 2 hours and 15 minutes to go from downtown Paris to downtown, London, and vice and versa,, and it takes 1 hour and 50 minutes to travel from London to Brussels. It is less expensive to go from London to Paris by Eurostar than to go from the London Eurostar railway station to the airport by cab.
- The French and British governments took no interest in funding the project: “not a public penny” said Margaret Thatcher, British Prime Minister in the 1980s. They don't regret having had no objection to the privately funded project.
The IT firm and the client
When a company contracts with an IT firm to put an enterprise resource planning in place, it is usually best when the issue is the same for both, but it is not. The issue for the IT firm could be the triple constraint, because the IT firm committed to the plan and they know what they should do (it is their job); they know how much time they should spend on the project and they know the resources they should use.
For the client, it is different. The issue is not only to set up the software, but also to use it and use it well. So, it is better to make some changes to the scope in order to set up the software so people will be able to perform with it. After a little experience on the project, the company might also realize that they selected the wrong software and the best thing would be to cancel the project as soon as possible.
The value of IT
IT investments require complementary investments to realize their benefits
Like John Thorp (1998, p18) says, most IT projects are transformation IT investments (others are capacity investments; they concern only IT infrastructure with no business impact). These transformation IT investments involve changes to the organization in order to realize their benefits. These IT investments require additional investments down the line. Setting up a customer center involves an investment in customer relationship management software, but the CRM software is only a part of the investment. Recently, a French bank completely transformed its relationship with customers and has invested 125 million euros in a new customer computing system. This IT part of the program counted only for 25% of the total cost of the project (500 M€).
The IT project should be a part of a program
We considered (2011) that the “IT project should not exist.” The so-called “IT projects” are business projects in reality. No organization aims to have more and more computing expenses, even though they all do. When they use more and more IT, companies intend to better perform somehow. IT sustains strategies (or should), but complementary investments in the organization are mandatory; otherwise, it is just a waste of money. The “IT projects” as they continue to exist finally, should at least be a part of a program. What counts is not project completion, but program completion, which is why this author shouldn't consider the success of an IT project, and the program success criteria themselves should be the business measures, not just the triple constraint.
Use the Right Success Criteria for Your Project
The importance of defining the project success criteria
Defining the project success criteria is not trivial, for two reasons. First, success criteria will shape the project team commitment; second, they are not easy to set up.
Setting up the success criteria of the project will align the entire team around common shared and detailed objectives. The contrary would be ambiguity among the team members about direction and priority or revisiting decisions and discussions again and again. Commitment depends on clarity. Great teams unite behind decisions and commit to clear success criteria.
Align success criteria to the business purpose
Identifying criteria is not easy because, as we've seen, success or failure depends on the stakeholder's point of view. An organization is a place where resources are allocated among stakeholders like shareholders, executives, employees, clients, suppliers, the government, and so forth. Doing a project implies allocating resources to a specific stakeholder. The purpose of a project could be one of these: increasing shareholder value, increasing security at work or employees' welfare, and embedding more functionality to the products to improve the customers' satisfaction. Every project is made for a certain stakeholder, and, of course, the project success criteria should be aligned with its business purpose.
Define your project success criteria
In order to define the success criteria of your project, I invite you to use Patrick Lencioni's model (2005, p 188): “the five dysfunctions of a team.” According to Lencioni, there are five reasons why teams meet with dysfunctions: absence of trust, fear of conflict, lack of commitment, avoidance of accountability, and inattention to result.
Trust is the first condition used to determine the right project success criteria. As we have seen previously, a project is a way to allocate resources between stakeholders across the organization. It means that probably not all stakeholders will benefit from the project. Some people might even suffer from the project. If the organization and especially the project team cannot accept that, it will not be possible to list the reasons for the project and, as a consequence, the success criteria will not be put on paper.
When Lencioni talks about fear of conflict as the second team dysfunction, he means conflict of ideas, not conflict between people, of course. It is important that all project members (those who take part in the early stage of the project) openly debate and disagree about important ideas such as defining the project success criteria.
Once the debate has taken place, a decision must be made, and all team members should commit to the decision, even if they somehow disagree with it. Discussions must be over once the success criteria have clearly been stated. The desire for consensus is a cause of lack of commitment. In order to support a decision, you don't need to agree with it, but just to know that your opinion has been considered. No matter if we are uncertain about the success criteria (as long as we know the objective): we might be missing some information at this stage. Better to decide on the success criteria than to waffle. They can be adjusted further along in the project, when we have more information.
Now that we have the success criteria, team members should be willing to call out their peers on performance or behavior that might hurt the success of the project. Team members should be accountable to each other for their actions.
Attention to the project success criteria
The last dysfunction would be to care about something other than the success criteria, which is why they should be used in the monitoring and controlling project processes complementary to time, scope, and costs. Some companies use executives to control projects when it becomes apparent too the executive committee that the project is no longer aligned with the success criteria. The person is not the sponsor of the project, but what they call an “exit champion,” because he or she can make the decision to terminate the project.
One should not consider that a project fails only because it is not on time, in budget, or in specifications. This narrow measurement of project success may even result in a biased assessment of project management skills. But contrary to what Atkinson (1999) writes, I think that success criteria depend on stakeholders' expectations only. The priority of projects and their success criteria should be defined in the project charter. Any change to the success criteria should be considered and managed as a change to the project baseline, because success criteria are parts of the baseline. The success of a project has nothing to do with the triple constraint (or “iron triangle”). Some companies have even reconsidered the prioritization of constraints in project management. For example, Disney considers safety, aesthetic value, and quality as priority constraints, according to H. Kerzner (2009, 143). Each project should have its own success criteria.
Atkinson, R. (1999). Project management: Cost, time and quality, two best guesses and a phenomenon, It's time to accept other success criteria, International Journal of Project Management(17)6, 337-342.
Buehler, R., Griffin, D., & Ross, M. (1994). Exploring the ‘planning fallacy:' Why people underestimate their task completion times. Journal of Personality and Social Psychology, 67, 366–381.
Cavarec, Y. (2011, May). IT projects should not exist. PMI® Global Congress, Dublin, Ireland
Channel Tunnel. (2012, February 21). In Wikipedia, the free encyclopedia. Retrieved February 23, 2012, from http://en.wikipedia.org/wiki/Channel_Tunnel
Dune, T., Roberts, M. J., & Samuelson, L. (1988). Patterns of firm entry and exit in U.S. manufacturing industries, Rand Journal of Economics, 19(4), 495–515.
Flyvbjerg, B. & COWI (2004). Procedures for Dealing with Optimism Bias in Transport Planning: Guidance Document. London: UK Department for Transport.
Flyvbjerg, T. Holm, M. S., & Buhl, S. L. (2002). Underestimating costs in public works projects: Error or lie? Journal of the American Planning Association, 68(3), 279–295.
Kahneman, D., & Tversky, A. (1979a). Prospect theory: An analysis of decisions under risk, Econometrica, 47, 313–327.
Kerzner, H., & Saladis, F. (2009). What executives need to know about project management. Hoboken, NJ: John Wiley & Sons, Inc.
Lencioni, P. (2002). The five dysfunctions of a team, San Francisco: Jossey-Bass.
Machiavelli, N. (1513). The prince. New York: Bantam Dell.
Malmendier, U. & Tate, G. A. (2003). Who makes acquisitions? CEO overconfidence and market's reaction, Stanford Research Paper No. 1798.
Morris, P. W. G., & Hough, G. H. (1987). The anatomy of major projects. Chichester: Wiley and Sons.
Planning Fallacy. (2012, January 3). In Wikipedia, the free encyclopedia. Retrieved February 22, 2012, from http://en.wikipedia.org/wiki/Planning_fallacy
Standish Group (1994) The chaos report Retrieved from www.standishgroup.com
Thiry, M. (2011, May). What your future looks like. PMI® Global Congress, Dublin, Ireland.
Taleb, N. N. (2007). The black swan: The impact of the highly improbable. London: Penguin.
Thorp, J., & DMR's Center (1998). The productivity paradox realizing the business benefits of information technology. Whitby, Ontario, Canada: McGraw-Hill Ryerson.
Wachs, M. (1986). Technique vs. advocacy in forecasting : a study rail rapid transit, Urban Resources4(1) 23-30.
© 2012, Yves Cavarec
Originally published as a part of 2012 PMI Global Congress Proceedings – Marseille, France
PMI research shows project teams that draw from an array of perspectives and skillsets deliver powerful outcomes.