Introduction
The future of project management involves being more successful more of the time. But what are the criteria for determining project success, and whose opinion about success counts? Does the project succeed if it meets the triple constraint, if it turns a profit, if it advances the state of the art, if it contributes to the overall value of a portfolio, or if it aligns itself with the organization's strategic objectives? Must the project satisfy all of these criteria, some of them, which ones? Is the project a success if it makes the customer happy, if it advances the project manager's career, if it pleases the boss, or if it pleases the lawyers? Whose opinion counts? This paper examines the widely differing viewpoints about project success and presents a best practice approach to improve the odds that projects will be viewed as successful. Industry surveys and references are used to illustrate the radically differing perspectives about project success and to make the point that the earliest phases of the project life cycle have the most dramatic impact upon the odds that projects will be viewed as successful. Opinions about success from the perspective of the customer, the senior executive, the line manager, the project manager, and others are examined. The paper draws on the guidance in the PMBOK® Guide and from respected authors about these early project phases, and it characterizes the importance of engaging the project stakeholders early and often in the project life cycle. Through the introduction of some interesting spy stories and a survey of CIA managers, the author uses experience from his 28 years in the project management business to describe the consequences of not paying enough serious attention to the subject of project success.
Following a characterization of the issues and consequences associated with the earliest project phases, the paper details an approach being used within the CIA's primary information technology organization to improve the odds that projects will, in fact, be viewed as successful. This approach tackles head-on the need to define the project success criteria early and obtain a broad consensus agreement from the large constituency of interested and affected parties. This approach involves revisiting both the project success criteria as well as the commitments to it periodically throughout the life cycle. Described are the steps in the Opportunity Planning phase of their project life cycle that deal with stakeholder analysis and defining the project success criteria. Also described are activi-ties throughout other phases in the life cycle to maintain focus on the success criteria. These Opportunity Planning phase steps are related to best practices in other industries.
The paper concludes with a summary of the major challenges associated with the introduction of this disciplined approach toward project success. Project team members', managers', and customers’ opinions within the CIA are summarized, as is the effect that this approach is having on the project success rate and the organization's project management maturity level.
The Problem
Precision Lacking
Let's get directly to the point: the big problem with assessing project success is that it is not precise. There is no consistent interpretation of the term “project success.” Many viewpoints exist from many people. And they change over time. This dynamic can often be the Achilles heal for a project. Without a dependable understanding of what constitutes success, the project is placed in the untenable position of being judged against differing criteria. This definitional moving target greatly increases the odds that the project will be viewed as something other than successful. The consequence of that harsh judgment is invariably one more failure statistic reported by research firms such as Standish, Gartner, Forrester, and others. While there is some comfort in the fact that information technology project success rates have risen in recent years, the fact still remains that only 28% of application development projects met the criteria for success (Standish, 2001).
Don't Ask
This lack of definitional precision is well known to many project managers, as is the fact that achieving some sort of agreed upon success definition prior to initiating a project is hard (Billows, 2001). So hard, that a popular practice has emerged to deal with it; and that practice has been to ignore it (Yourdon, 2001). It is amazing how many intelligent, experienced project managers never ask these two fundamental questions at the beginning of their projects: Who has the right to declare success? And what are the criteria that will be used to determine success or failure?
Exhibit 1. Historical Success Definitions
CIA Example
In the late 1950s the CIA responded to the need to learn more about the Soviet intercontinental ballistic missile program by undertaking a project to design, build, and fly super high altitude spy planes equipped with cameras (Richelson, 2001). This project produced U2 spy planes capable of sustained flight at never before achieved altitudes, a tremendous first of its kind technical success. But the project was done without a great degree of planning and without many project controls. This was a questionable approach at best when compared to mature project management practices, and perhaps even a failure when compared with today's project management best practices. However, the timeline was quite short, taking only about one year from the start to the actual deployment. This schedule was so remarkably short that awards were given to those involved. But the costs were higher than tolerable for many of those involved and high enough for some of those in an oversight capacity at the time to express opinions about the project in terms other than successful. These U2 spy planes repeatedly flew over enemy territory and provided the United States with invaluable intelligence; many credit the U2 project with the peaceful resolution of the Cuban missile crisis when President Kennedy used photographs taken from this spy plane to demonstrate that the Soviets had been secretly constructing missile bases in Cuba. Preventing what could have been a world war can undoubtedly be viewed as a success, a huge success. But it was this very same U2 that also brought the United States a huge international embarrassment when the spy plane piloted by Francis Gary Powers was shot down over the Soviet Union and that evidence was used to demonstrate publicly that the United States had not been truthful. This incident, which occurred at the height of the Cold War, was a hurtful failure for the United States. The technology, the industrial capability that was brought to bear developing the U2, and the strategic inertia that was created by the U2 project all served to propel the United States into a sustained long-term program for high altitude intelligence collection that continues to this day.
So, was the U2 project a success or a failure? If we look at the abbreviated scorecard the U2 project was a huge success from the perspective of the technology, the schedule, and the photographic intelligence. In fact, within the CIA, the U2 is often referred to as one of its greatest success stories. But when we take the perspective of project management best practices, costs, and the international embarrassment, the U2 could be considered as less than successful. The answer to the question about the U2 being a success or failure is both. It was both a success and a failure. And the answer depends on what criteria are being used and whose opinion is being considered.
What Are the Criteria?
Much Known
Much is known about criteria for project success. The PMBOK® Guide, project management practitioners, and respected authors have given considerable emphasis to this subject. The importance was underscored in 1986 when the Project Management Institute (PMI®) devoted its Annual Seminars and Symposium to this topic. This has resulted in a rich understanding by many about what constitutes project success. The challenge, however, is to deal with the fact that these viewpoints differ. And they differ in numerous and significant ways. Consequently, today's project manager is confronted with a diverse and often conflicting array of choices about the success criteria to be chosen when initiating his or her project. And this diversity and uncertainty survive the project life cycle only to reappear at the conclusion when judgments are being made about whether the project did, in fact, succeed.
The History
Over the years the definition of project success has changed (Kerzner, 2000). In the 1960s, the early days of project management, success was measured entirely in technical terms. Either the deliverable product worked or it didn't. During the 1970s that narrow definition was expanded to encompass completion on time, within budget, and at an acceptable level of quality. This has become known as the triple constraint and has been widely used as the basis for much of the project management industry. During the 1980s further expansion took place to include the criteria of being accepted by the customer. And during the 1990s still more criteria were added, having to do with not disturbing the main work flow of the organization and with not changing the corporate culture.
Two Components
Two distinct components have been identified which do an excellent job of clarifying the definition of project success: project management success and product success (Baccarini, 1999). Project management success focuses on the project process and, in particular, the successful accomplishment of the triple constraint of cost, time, and quality objectives. Product success deals with the effects of the project's final product. Conceptually, the determination of project management success disregards product success, e.g., a product has been managed efficiently but eventually does not meet customer or organizational expectations. The focus of project managers on success is highlighted by research on IT projects, whereby project managers interpreted a successful project as one meeting budget and schedule, i.e., project management success; while users placed greater emphasis on the meeting of requirements such as response times and reliability, i.e., product success (Wateridge, 1995). This indicates that “project managers are focusing on the short-term criteria relating of the project process and concentrating on meeting time and budget constraints … as opposed to the longer-term criteria relating to the product, such as delivering a system with which the users are happy” (Wateridge, 1998). A similar perspective identifies multiple layers or interwoven aspects of the project life cycle (Forsberg, Mooz, & Cotterman, 2000). The business layer contains the business case and related opportunity feasibility and assessment factors, while the technical layer contains the more precise factors such as requirements collection and system specification. Success in this case is tied to the simultaneous satisfaction of factors from both layers.
Portfolio Management
Techniques for portfolio management have been identified which can be most helpful in establishing meaningful project success criteria. Portfolio management represents one more source of viewpoints about how to define project success and make project selection decisions. It has emerged as a reasoned approach to making project decisions linked to business policy and organizational strategy (Dye & Pennypacker, 1999). Portfolio managers make the key decisions about which projects are initiated and become part of the portfolio, and they make decisions about which projects are terminated and consequently exit the portfolio. These decisions are based less on the individual merits of one particular project and more on the project's contribution to the overall success of the portfolio. Although the specific decision or success criteria may vary these techniques commonly include the need for the criteria to be explicit and measurable, be used and adjusted over time in a disciplined way, and be communicated clearly to all parties involved in the project (Frame, 1994).
Levels in Organizational Hierarchy
How we view project success is linked to the position we hold in the organizational hierarchy. It has been long understood that the view from the top is quite different from the view from the lower levels (Archibald, 1976). Project managers tend to view project management quite personally, often with little or no distinction between their performance and the performance of the project. Success for them can be measured by career advancement. On the other hand, functional or mid-level managers tend to see project management process in a fairly negative light. Often it can be viewed as a dangerous threat to their established authority and organizational domain, as evidence of implied deficiencies or failures in the traditional functional organization, with reflections on the functional managers personally. For them, increased reliance on project management process equates to decreased reliance on their authority. They would have difficulty declaring success if it meant that they could not retain full power and control. Top managers, however, tend to see project management in various and sometimes conflicting but usually positive ways. They see it as a means to an end, as a good way of motivating people toward achievement of specific objectives. Success for them is often associated with meeting the delivery schedule.
Informal Survey
Even people at the same level in the organizational hierarchy have considerable difficulty with the subject of project success. An in-formal survey recently conducted of 50 line managers in the CIA produced results with a level of inconsistency and diversity reflective of the problem. As a group these line managers all worked in a project based part of the organization, they were all relatively experienced with the discipline of project management, they were all involved with similar technology, and they all had responsibility for the management of project managers and their respective project teams. Yet when asked to list the five most important characteristics of a successful project they as a group could not narrow the list down to anything shorter then about 15 items. Their expressed frustration as individuals was with the fact that they had expected that the group definition of project success would agree closely with their individual definitions. The impact of their predicament became obvious when the question was asked, “If agreement to the definition of project success can not be easily achieved within this group—how can the project manager fare any better?” It was an excellent learning point, which clearly demonstrated both the difficulty and importance of dealing with project success. And it reinforced the important impact line managers can have on creating an environment for project success (Graham & Englund, 1997).
Exhibit 2. Punch Line
The Solution for the Future
Key Elements
We've seen that a great deal of diversity exists in terms of what can be considered to be project success criteria. Words such as ambiguity, uncertainty, disagreement, and conflict have been associated with attempts to define project success. The definition can include many elements beyond the triple constraint such as being accepted by the customer, not disturbing the main work flow of the organization, and not changing the corporate culture. We've seen that it can include the two distinct components of project management success and product success. And we've seen that it can be highly influenced by the level in the organizational hierarchy, the mix of the portfolio, and the uncertainties of line managers. Most importantly, we've observed that a predominant method for dealing with this definitional diversity is through avoidance.
The future of project management involves being more successful more of the time. Avoidance of the project success issue should not be an option. That means that there will need to be an intentional concentration early in the project life cycle on establishing the success criteria. In turn, that means examining the project life cycle. In some project life cycles or methodologies the success criteria can be formulated during the initial phase; other life cycles refer to this period as the feasibility phase, while still others refer to it as the concept phase. In terms of the PMBOK® Guide, this means focusing attention on the success criteria in the planning phase. Scope initiation and scope planning are specifically identified in the PMBOK® Guide as processes by which decisions are made about the project success criteria. Whatever the life cycle, including activity early on that defines the success criteria for the specific project being initiated, will be key (ITRB, 2001).
Exhibit 3. Opportunity Planning
Opportunity Planning
A leading CIA information technology organization has been using a structured and disciplined process, Opportunity Planning, which is based on the best practices of defining the project success criteria. Opportunity Planning has been formally inserted as a phase near the beginning of the life cycle. Project teams are following Opportunity Planning to guide their efforts to plan projects, while line managers and others follow this process to guide their interactions with the project teams. Mutual expectations exist about what needs to be done, how it will be done, and who will do it. All have received hard copies of this process, all have received templates for the associated documentation, all have access to it as an online tool, and all have participated in training regarding its use. It details a common way for those involved to plan their projects. And it details a common way to define the project success criteria.
The Opportunity Planning phase immediately follows the decision that is made to accept the opportunity, and it is followed by project execution and deployment (O'Brochta, 2001). Opportunity Planning is based on following a series of steps that are sequenced between control gates. Decisions and written deliverables are required to pass beyond a control gate. During Opportunity Planning a stakeholder analysis is performed, the success criteria are established, risks are planned for, and a project plan is developed.
Stakeholders Are Important
The PMBOK® Guide makes a strong case, which is supported by current literature, for stakeholder involvement, particularly as it relates to making the decisions about the project success criteria (Smith, 2000). Stakeholders as defined in the PMBOK® Guide are individuals and organizations that are actively involved in the project, or whose interests may be positively or negatively affected as a result of project execution or project completion. The stakeholder mix will most likely be situational, varying by organization and by project. For some the mix will include the project manager, customer, performing organization, and sponsor. For others the stakeholder mix may include owners and investors, suppliers and contractors, team members and their families, government agencies and media outlets, individual citizens, lobbying organizations, and society at large.
Within the Opportunity Planning process, the decision about which stakeholders to include in the mix is determined by a stakeholder analysis. Stakeholders are identified and classified according to their interests, impact level, and relative priority. The results of this classification are displayed on an importance-influence diagram whereby the stakeholders with the combination of the most importance and the most influence are identified and then interviewed. The circle of people and organizations included in these interviews is typically broad. Those interviewed often include actual end users and customers, their management, the primary provider for their existing suite of information services, the organization responsible for providing operations and maintenance support, the management responsible for overseeing the assessment of the opportunity, and the policy makers. During interviews needs and wants are captured in writing in a Requirements Document to be used later in the process. On occasion interviews conducted for one opportunity can, if still current, be used for a similar or related opportunity, thereby reducing the number of interviews.
Success Criteria Document
Projects that follow Opportunity Planning produce a formal document that captures the project success criteria. The most important aspect of the project success criteria document is not so much it's specific content, but the fact that it exists at all. In fact, the guidance to the author that is contained in the document cautions “success criteria are by their nature highly dependent on the circumstances for the individual project.” With that as a given, the author is advised to base all of the success criteria on the results of the stakeholder analysis and to ensure that each of the criteria is measurable. Suggested categories of success criteria within the document include project performance, cost, schedule, and security. Within the CIA this is often referred to as the “quadruple constraint” much in the same way the industry at large uses the term “triple constraint.” Additional criteria are suggested which include customer satisfaction, degree of improvement compared to current conditions, amount of expected use, strategic alignment, architectural compliance, and political good will. No attempt is made to be prescriptive in the sense that the specific suggested success criteria will be appropriate for all projects in all situations. Rather, the point of the success criteria document is to ensure that the success criteria for this specific and unique project are captured and agreed to. Names used by other organizations for this type of document might include Initiative Business Case, Project Objectives, or Project Balanced Scorecard (Stewart, 2001).
Exhibit 4. Success Criteria Document
Revisit
During the project life cycle the stakeholder viewpoints can change or the stakeholder mix can change. One particularly successful CIA project developed a computer tool that enabled searches to be conducted of documents in languages that the user did not understand (Zakaria, 2001). As time went on and delivery of the initial language sets took place, new stakeholders emerged with new language needs. The definition of success in this case was evolutional.
Revisiting the stakeholder analysis when this occurs ensures that the tight coupling between the project and the stakeholders is maintained. Within the Opportunity Planning process, once the Success Criteria Document is prepared and used as the basis for agreement among the key stakeholders, it remains an active part of the life cycle. Its contents are revisited at key control gates or milestones throughout the life of the project. And most importantly, the level of stakeholder commitment to these success criteria is revisited as well. During the project execution and deployment phases, stakeholders are members of the teams that are responsible for project review decisions. They express their confidence about whether the success criteria are being met by their vote. And at the conclusion of the life cycle the stakeholders are formally surveyed to assess their degree of satisfaction against the Success Criteria Document. The results of this comparison is documented and used as a required part of the planning in the life cycle for future projects.
Challenges
The organizational elements within the CIA that are using the Success Criteria Document are using it as part of a total life cycle that was formally adopted during the last couple of years. The development and adoption of this formal process was driven from throughout the organization by the desire of the project teams for procedural consistency and by the organizational management's desire for more predictable project results. It was requirements-driven. While this underpinning of broad support did have the effect of minimizing the challenges associated with implementation, some hurdles did arise.
An initial consequence of following the Opportunity Planning process and producing a Success Criteria Document for each project was delay. The time required to insert these efforts was not trivial, especially when compared to the long-standing practice of minimizing or even avoiding this type of effort. In some cases this delay was used as a justification for discarding this new discipline, but by and large those involved remained faithful. Their perseverance did pay off. The people and the organization quickly became skilled and experienced and found that within each of their areas of business a clear and repeatable pattern emerged. This pattern, due to the fact that the same people were repeatedly working on projects in the same line of business and technology, resulted in considerable efficiency. The benefits of business specialization took effect. As a result, Opportunity Planning and the development of success criteria for each project became more routine.
Exhibit 5. Current Project
Conclusions
As part of the overall life-cycle process, the focus on the definition of project success has already been found by the project teams to reduce the uncertainty associated with getting projects started. It has also helped to make the efforts of project teams more consistent with each other, while simultaneously increasing clarity for the management reviews of the projects. Project management maturity has been measured and found to be greater for the organizational elements where these practices have been used for longer periods of time. And some higher risk opportunities that might have become projects in the past are being avoided; as a result, project success rates are rising. Both project team members and management speak openly about the benefits. Efforts are currently under way to greatly expand the use of these best practices to all information technology projects within the CIA and to integrate these practices into the formal management and governance process.