Defining project success

a multilevel framework

NICTA, Australian Technology Park, Sydney, Australia

Introduction

In the drive toward formalizing project management as a distinct discipline, there has been much discussion on the nature and definition of project success, but no consensus has emerged. This paper aims to contribute to the development of the field by synthesizing the seminal literature into a multilevel framework of project success that has wide application in practice. The proposal is supported and illustrated by reference to information systems (IS) development projects, but the fundamental framework is relevant to most, if not all, project applications, including product development, engineering, construction, aerospace, and defense.

Projects are discrete but multidimensional activities that serve as vehicles of change. They have a checkered history of performance (especially IS projects). However, it is not easy to develop a consistent view of the problem because of this multidimensionality. Empirical studies tend to use different definitions of project success, making comparison difficult. In the literature, project success variously refers to “on time, within budget, to specification” completion; success of the product produced; or success in achieving the business objectives of the project. Also, these measures are often contested, making it difficult to determine whether there is a problem at all (Sauer, Gemino, & Reich, 2007). A further complication is that, like quality, success is perceptual and perceptions vary with the stakeholder's perspective and the passage of time since project completion. Despite these challenges, unlocking the problem of defining project success is critical for progressing project management research and extending the knowledge base of this nascent discipline.

Much work has already been done in researching project success but the main objective of that research is often to unlock the drivers of project success rather than establish a common framework for determining whether a project is a success. The framework proposed in this paper comprises five levels of performance criteria that permit assessment of a project from multiple stakeholder perspectives at different times after project closeout. Three of these criteria (Levels 2 to 4) comprise criteria commonly discussed in the research literature (project management success, product success, and business success); another criterion (Level 5, strategic success) extends views that suggest consideration of broader, future benefits; and a final criterion (Level 1, process success), not discussed in the project management literature as a success criterion, supports learning and development in project application domains. Based on this framework, project success is the highest level achieved at any point of reflection.

The paper offers three main contributions. First, it formalizes two additional success criteria, one conceptually closer to the project action and the other further away (Levels 1 and 5). Second, it structures these and other extant criteria into a multilevel framework and assessment approach that has practical utility for determining project success. Third, it provides a rationale for using the framework that contributes to overcoming the problems associated with defining project success by aligning success determination to a common reference framework.

The paper is structured as follows. First, the literature is reviewed to position the problem, identify success criteria, and uncover gaps in the research-practice nexus. Then the proposed multilevel project success framework is described. Application of the framework is then illustrated using information systems case studies. The paper concludes with a discussion of the contribution and directions for future research.

Literature Review

Project management has a rich history (Cleland & Ireland, 2007; Kloppenborg & Opfer, 2002; Morris 1994) but is still emerging as an independent discipline with its own theoretical foundation (Koskela & Howell, 2002; Morris, 2002; Sauer & Reich, 2007; Söderlund, 2004). In particular, it lacks a common measure of project success and failure.

Since a project is usually a means to an end rather than an end in itself, it is reasonable to want to know if the project is successful, whatever the end might have been. Herein lies a difficulty. The success of a project can be determined from the perspective of the means (the project itself) or the end (what it was intended or expected to accomplish) depending on the interests of the stakeholder. Furthermore, regardless of means or ends, expectations of what the project was to achieve and perceptions of whether it achieved them often vary among stakeholders. This makes determination of project success highly contingent upon the expectations and perceptions of different stakeholders, and when the assessment is made (de Wit, 1988).

Most projects have multiple stakeholders with different views on the project's purpose and different expectations of what the project must achieve (Lyytinen & Hirschheim, 1987). These stakeholders might include the people who originally identified the need for the project, those who fund the project, those who stand to benefit from the project, the people who are impacted by the project and its outputs, the project team members, and the people who have to oversee the project. Each has a vested interest in the project's outcome, with different expectations and perceptions.

If we assume that a project is an end in itself, then its success can be determined at closeout stage. However, if it is a means to an end, then its outcome can only be measured at some time after the formal project has completed. This permits the entry of other events and influences into the perception of project success that may not be a realistic reflection on the achievements of the original project.

To unlock this issue of defining project success, we review how the problem has been approached in the literature as input to proposing an integrated multilevel project success framework as a way forward.

Project success has attracted much attention in the research and practice literature. Three main streams are found. The first and dominant stream aims to identify the factors that practice suggests are likely to contribute to project success, project failure, or project risk (e.g., Baker, Murphy, & Fisher, 1988; Cooke-Davies, 2002; Pinto & Covin, 1989; Pinto & Slevin, 1988b; Schultz, Slevin, & Pinto, 1987; Slevin & Pinto, 1986). Typically, this literature produces prescriptive lists of critical success factors, failure factors or risk factors that project managers and governance bodies should take account of to ensure a positive project outcome. The value of this research stream is that it identifies important preconditions and drivers of project success, but it provides no explicit definition of project success itself (although the factors identified may indirectly point to relevant criteria).

The second stream focuses on identifying other contingency variables that might impact project outcomes or require specific management intervention to mitigate any potential negative effects.1 Some researchers call these variables “dimensions” of project success. They include project size (Yourdan, 1997), project type (Pinto & Covin, 1989; Shenhar, Tishler, Dvir, Lipovetsky & Lechler, 2002), life cycle stage (Pinto & Mantel, 1990), project management complexity (Shenhar & Wideman, 1996), and strategic versus operational mindsets (Schultz, Slevin & Pinto, 1987; Shenhar, Poli & Lechler, 2000). The value of this research is that it identifies additional project variables that may have a critical impact on project success, depending on the context of the project and how the variables are managed. Again, however, this stream does not explicitly define measures of project success.

These two streams are often considered in conjunction with a third stream, whose main interest is in defining the criteria by which a project is judged to be a success or a failure. The three streams are interrelated in that the first two are concerned with how to achieve project success, while the third is concerned with the measures against which success is judged. Knowing how success is defined is a necessary precursor to determining where and how project effort should be focused to meet performance goals; and knowing where to focus project management effort is guided by an understanding of the drivers of project success and failure. Having a common definition of project success also facilitates agreement on whether, in the face of disparate interests and perspectives, success has been achieved. However, in the past, there has been an imbalance of attention in these streams of research. Research and practice have tended to focus on “how to do it right” (the first two streams) at the expense of reaching consensus on what “right” is (the third steam).

Some researchers suggest that success criteria should be project-specific and therefore determined by stakeholders at the start of each project (e.g., Baccarini, 1999; Nelson, 2005; Turner, 2004; Wateridge, 1998). This view has considerable merit because of the broad range of project types, project objectives, and other variables that can contribute to project outcomes. However, there is also a role for a common reference framework to enable project success to be discussed in a uniform way and to provide a standard benchmark by which project outcomes can be compared (Pinto & Slevin, 1988a), especially within the same discipline.

Several surveys of project success research already exist in the literature (e.g., Cooke-Davies, 2004; Jugdev & Müller, 2005; Pinto, 2004; Shenhar, Dvir, & Levy, 1997; Wateridge, 1998). Furthermore, Baccarini (1999) summarizes characteristics of project success criteria. The present review is not intended to be exhaustive in uncovering either the range or depth of contribution of the literature to this discussion. Rather, it focuses on identifying success criteria that the literature suggests would present difficulties in practice if excluded from a definition of project success.

Project Management Success

The classic criterion from practice is a measure of the immediate performance of a project against its main design parameters—schedule (time), budget (cost), scope, and/or quality—which the literature tends to call a measure of project management success. This definition was already established in the earliest discussion of projects in the management literature. Gaddis (1959) defined a project as “an organization unit dedicated to the attainment of a goal—generally the successful completion of a developmental product on time, within budget, and in conformance with predetermined performance specifications” (p. 98). In the three-element form, this criterion is variously called the triple constraint, iron triangle, or three-legged stool of project management. Other variants include all four elements as the project diamond or four-legged stool. Scope is less clearly defined than time or cost, referring to the extent to which the main deliverable was completed against specification or whether all intended activities and phases of the project were completed. Quality is often assessed, post hoc, against established industry or subjective criteria. The conventional approach is that an assessment of performance is made in a post project review based on whether the project was completed “on time, within budget and to specification.” If each was achieved within a narrow range of tolerance then the project is deemed a success. This criterion is of particular interest to stakeholders with vested interests in the project vehicle itself, such as the project manager, project team, and project governance stakeholders.

This classic criterion remains the most widely used measure of project success. Its main value is in offering a simple, direct measure of performance of a project and the project management expertise applied to complete the project within the bounds of the most immediate design parameters (time, cost, and scope). However, it has major limitations. Most critically, it focuses on the means rather than the ends of the investment from the organizational perspective. It takes limited or no account (depending on how scope is defined and measured) of whether the main project deliverable fulfilled the purpose for which it was intended and whether the objectives of the project's investors were achieved. For example, it is not unusual, especially in IS projects, for a project that is late, over budget and/or under-delivered against specifications to be declared a success, because it still delivered a benefit to the client/users and/or to the investing business. This suggests the need for two additional success criteria: measures of project deliverable or product success and business success.

Product Success

Drawing on the IS literature, several researchers implicitly argue that project success is a function of the success of the information system produced by the project (e.g., Ballantine et al., 1996; Delone & McLean, 1992, 2003; Seddon, 1997; Seddon, Staples, Patnayakuni, & Botwell, 1999). Completing the main project deliverable “to scope” (specification) may not be a valid or sufficient measure of project success if the deliverable is not also accepted and used by the intended client/end-user and/or does not provide sufficient benefit to them. In the case of information system deliverables, this success criterion might comprise measures of information quality, system quality, service quality, intention to use, actual use, user satisfaction, and net benefits (DeLone & McLean, 2003). The same concept, however, can be applied to buildings and other civil infrastructure, for example, produced by construction and engineering projects (Baccarini, 1999). Clients/users have different interests and expectations of a project to triple constraint stakeholders. They center on fitness for use and other benefits associated with improvements in the nature and conditions of their work. A project can succeed in delivering an information system “on time, within budget and to specification” but fail to gain user acceptance or use of the system. It is well-recognized that this can occur, for example, when a system specification lacks adequate user input to its definition and/or when user requirements change due to evolving business circumstances.

This view of project success is fundamentally consistent with Pinto and Slevin's (1988a) seminal model of project success and Kerzner's (2003) definition. Pinto and Slevin (1988a) modeled project success as comprising two components: success of the project itself, as indicated by time, cost, and performance subcomponents, and client success, as reflected by use, satisfaction, and effectiveness of the project in benefiting intended users. Similarly, Kerzner (2003) defined project success as completion within time, cost, and specifications (the traditional triple constraints), as well as with minimum or mutually agreed scope changes and acceptance by the client/user (what I have called product success). Kerzner added two additional components: completion without disturbing the main workflow of the organization and without changing the corporate culture. The intent of these components is not to argue that projects should not be vehicles of change within the organization but an acknowledgement that projects execute within an existing operational organizational context with established values and norms of behavior. This is consistent with the view of a project as a discrete change activity within an organization.

Business Success

In the case of the second additional criterion, a measure of business success, de Wit (1988) argued for distinguishing between project success and project management success. The latter is determined by the triple constraints of time, cost, and quality, but the former is a measure of the degree to which the project objectives are met and benefits accrue to the investing organization. Other researchers refer to this additional criterion as the business or organizational objectives criterion (e.g., Ballantine et al., 1996; Shenhar, Dvir, Levy, & Maltz, 2001). Describing these objectives as business or organizational objectives rather than project objectives begins to resolve the problem raised by de Wit (1988) of multiple stakeholder objectives relating to a project. Simplistically, project objectives relate to the goals in the project plan while business or organizational objectives relate to the goals in the business plan.

Taking information systems, ultimately, businesses do not invest per se in a new computer system for the right system to be installed on time, within budget, to specification, and the satisfaction of users. Instead, they aim to solve a particular business problem (albeit in a timely, cost-efficient, and effective manner). If the project does not deliver an acceptable solution to that problem then investment stakeholders are likely to view the project as a failure. Naturally, the business success criterion also permits the perverse possibility encountered in practice of a project failing on project management and/or project deliverable criteria but still achieving business objectives in some acceptable way and, therefore, being considered a success. This reinforces the counterintuitive view that project management success and even project deliverable success are neither necessary nor sufficient for project success.

Not all researchers support this three-tiered view of project success, as presented so far. For example, discussing information systems failure (rather than success), Lyytinen and Hirschheim (1987) argued for the notion of expectation failure (“the inability of an IS to meet a specific stakeholder group's expectations,” p. 263) in favor of what they called process failure (the IS is not designed within time and cost constraints), interaction failure (the IS is not used), and correspondence failure (the IS does not match goals). Stakeholders are defined as “all those claimants inside and outside the organization who have a vested interest in the problem and its solution” (p. 261). Their view is that failure (success) is the embodiment of a perceived situation, and stakeholder perception/expectation is a superset of the other three criteria. This view is inconsistent with a definition of project success based on a single criterion, but it is consistent with a multilevel project success framework that permits multiple stakeholder interests at different levels of perception and at different timeframes as proposed in this paper. It is also consistent with de Wit's (1988) view that stakeholders have many different objectives of a project. This leads us to a fourth success criterion.

Strategic Success

De Wit's (1988) concerns about multiple stakeholder objectives are further partially addressed by criteria that assess project success beyond direct organizational benefits. These include benefits that favorably position the organization for future opportunities (Shenhar et al., 2001), and benefits that accrue to the stakeholder community beyond the investing organization (Atkinson, 1999). In the proposed framework, these considerations are generalized as a strategic success criterion. This criterion represents the highest level of benefit achieved by a project, despite the possibility of failures against lower level criteria, as recognized by external stakeholders, such as investors, industry peers, competitors, or the general public, dependent upon the nature of the project.

Take an example from engineering and construction, the Sydney Opera House. The building was completed 10 years late, nearly 15 times over budget and significantly functionally constrained (its small opera stage and buried orchestra pit are well-known), yet it has served Sydney's performing arts patrons successfully for 35 years, has been a boon to its owners, and the appeal of its distinctive architecture has attracted international attention, including recognition in 2007 as a UNESCO World Heritage Site. Typically, few projects achieve strategic success. However, this criterion enables the perceptions of a broader community of stakeholders than those in the investing organization to be represented, and a broader range of benefits than might have been intended. It is the ultimate confirmation that a project delivered an outstanding end result.

In addition to engineering and construction, this criterion is particularly relevant to information systems projects. In the 1980s, the concept of strategic information systems emerged around the use of information systems to gain competitive advantage and other strategic benefits (Porter & Millar, 1985). While a number of strategic information systems are identified in the literature (e.g., Ciborra & Jelassi, 1994), it is also recognized that achieving strategic information system status is not simply a matter of design or intent (Emery, 1990). Its success depends on its effectiveness in the marketplace and is dependent on the perceptions of competitors and other industry stakeholders external to the investing firm.

Process Success

A final project success criterion is not recognized in extant project management literature. It responds to the need to consider technical and managerial processes associated with project management that are important at different times throughout the project life cycle. Consequently, this is a lower level criterion than the project management criterion discussed above. Consistent with the aims of quality management (such as Deming's Quality Management System, Total Quality Management, Motorola's Six Sigma, Lean, and ISO 9000 standards), this criterion provides a basis of focus on continuous improvement of critical discipline-specific in-project processes.

In the case of information systems, for example, the systems and software engineering literature has a strong interest in the processes that underpin successful information systems development and deployment. For example, software process improvement (SPI) is a key component of software quality accreditation (Chrissis, Konrad, & Shrum, 2007). Key related processes include software development methodologies, configuration management, risk management, change management and quality control within the context of project and technical governance mechanisms. Most post-project reviews include consideration of these processes in determining project management success. Reviews typically consider whether the right processes were chosen and effectively applied when needed, and whether they were appropriately aligned and integrated to facilitate achieving the objectives of the project.

However, there is no explicit criterion in the project success literature to focus attention and learning on these critical in-project processes. The absence of such a criterion makes it difficult, for example, for a stakeholder outside of the project to know whether a project was late because of poor schedule management or some other embedded process within the project. A lower level criterion would facilitate a finer grained examination of performance that could directly support incremental learning and improvement of contributing processes.

Project disciplines other than information systems (such as product development, construction, aerospace, etc.) have their own sets of critical in-project processes that could similarly be assessed independently of their net effect on the overall project management result with a success measure at this level of analysis.

The above literature-based analysis has focused mainly on identifying different project stakeholder perspectives. However, these perspectives also correlate with time; that is, with when the assessment is made. For example, if the assessment is made immediately after project closeout, there is little more in extant project success definitions than the traditional triple constraint criterion upon which to base the determination. After the passage of time, however, assessment becomes possible against the criteria of the other interests and perspectives. Pinto and Slevin (1988a), for example, argued that

there are definite benefits involved in waiting until after the project has been transferred to the clients for whom it is intended before assessing project success. By waiting until the project is up and functional, we are better able to understand the impact of the external organizational factors, such as value and organizational effectiveness (impact). (p. 70)

Viewed alternatively as a financial investment, a “successful” project may show increasing returns on investment over time, so perceived value and success may increase with time. As with any investment, the derived value (return on investment) can be assessed at multiple points in time after the initial investment and at multiple levels of analysis (that is, from different perspectives).

This contrasts with the current tendency to determine project performance at only one point in time (hence the argument to delay the assessment until the most appropriate time). This latter approach creates the dilemma that if project performance is determined at one point in time, which point is “best”? Determining this point has been the preoccupation of much of the extant literature on project success. The proposed multilevel framework, which we now consider, overcomes this dilemma.

Multilevel Project Success Framework

An alternative approach to the problem of defining project success is a framework that enables success to be determined at key milestones at different times after project closeout and from different stakeholder perspectives (Figure 1). Key milestones in this spectrum relate to the project itself (the processes used and their effectiveness in delivery the project within the major design constraints), the product or main deliverable produced by the project (its fit to specifications and purpose as well as acceptance and use), and the organizational benefits returned from the investment (achievement of business objectives and the generation of strategic value). These milestones represent five levels at which project-related performance can be formally or informally assessed. Levels 2 to 4 reflect criteria commonly found in the literature. Level 5 is implied by some researchers but made explicit in this framework, while Level 1 incorporates a measure of technical performance found in disciplinary domains to promote learning and development at the tactical level in the design and execution of projects.

Five Levels of Project Success

Figure 1. Five Levels of Project Success

This approach enables success to be determined and periodically re-determined as benefits accrue from the project over time. It also enables stakeholders to progressively map success to perceptions of higher derived value from the project as benefits accrue. Based on this framework, project success is defined by the highest level of benefit achieved by the project at any point of reflection. This makes it possible, even non-controversial, for a project to fail at a lower level of assessment but still succeed at a higher level of perceived return from the project.

Each level is briefly described, following, and detailed in Table 1.

Level 1 – Process success. Every project discipline has generic and project-specific best practices that are critical to successfully completing a project. Even generic processes such as project management and risk management have discipline- and domain-specific best practices. Others will be exclusive to the discipline. Determination of success at this level considers the appropriateness of the processes used, their alignment with the project's purpose, and their integration and effectiveness in contributing to the project outcomes. As with the other levels, analysis here provides feedback to the project team and organization for learning and improvement for subsequent projects.

Level 2 – Project management success. This is the traditional criterion of project success, determined on closeout against key project design parameters such as the project schedule, budget and some performance expectations such as completing all planned stages and activities. Note that scope at this level refers to project scope, not product scope (the latter is a component of the next level).

Level 3 – Product success. This level considers the success of the major deliverable from the project. Clearly, what this is will vary with the project discipline and the specific project (e.g., an information system, building, submarine, road, or some form of service deliverable). It includes measures relating to the deliverable itself (such as its match to specifications, requirements, and quality expectations) and to client/user satisfaction (such as product acceptance, use, and effectiveness).

Level 4 – Business success. Success at this level is accounted as accrual of positive net benefits to the organization from the project. It may also include an assessment of the organizational contribution to the project's outcome. Consequently, measures will typically include the degree to which the project met the goals and objectives that motivated the investment approval (which are usually specified in the business plan), and whether the expected benefits were realized. They may also include consideration of the effectiveness and contribution of corporate governance to the project. Finally, assessing net benefits will also include any unintended benefits and negative impacts that arose from the investment.

Level 5 – Strategic success. At this level, organizational benefits are assessed by external stakeholders such as investors, competitors, industry analysts, or regulators, rather than company insiders. Success at this level derives from net improvements in industry position, business growth and development, competitive advantage, and/or other strategic gain. Strategic success may be planned or emergent.

The empirical indicators in Table 1 are adapted and extended from Shenhar et al. (2001). They are generic pointers to project-specific criteria.

Table 1. A MultiLevel Framework of Project Success

Level Success Criterion Description Empirical Indicators
1 Process Discipline-specific technical and managerial processes, methods, tools, and techniques employed to achieve the project objectives. Technical and managerial processes were:
  • Appropriately chosen for the purpose
  • Aligned with the project objectives
  • Integrated with each other (as appropriate)
  • Effectively implemented
2 Project Management The project design parameters or objectives. Here “scope” refers to the intended scope of the project (e.g., to specify, build, test, and implement a new system), not the scope of specifications of the main project deliverable.
  • Schedule met
  • Budget not exceeded
  • Project scope achieved
3 Product The main deliverable(s) from the project. The nature of the deliverable(s) will be discipline-specific. For example, it might be a product, system, building, bridge, airplane, rocket, or a service of some kind.
  • Specifications met
  • Requirements met
  • Client/user expectations met
  • Client/user acceptance
  • Product/system used
  • Client/user satisfied
  • Client/user benefits realized
4 Business The business objectives that motivated the investment. That is, what the business wanted to achieve from the investment.
  • Objectives met
  • Business case validated
  • Business benefits realized
5 Strategic Business expansion or other strategic advantage gained from the project investment, either sought or emergent.
  • Business development enabled
  • External stakeholder/competitor recognition
  • Competitive response generated

In using the multilevel framework, five basic “rules” apply:

  1. Project success may be determined at successively higher levels as time passes from project closeout.
  2. A project may succeed at a level but “fail” (or, more correctly, not succeed) at one or more subordinate levels. For example, it is possible to succeed at Level 4 (business success) but not be successful at one or more of the lover levels (Levels 1 to 3). That is, project success at one level implies nothing about the project's performance at other levels.
  3. Project success is determined by the highest level achieved at any point in time. For example, if a project is determined to have succeeded at Level 4 (business success), the project's success status is that it is/was successful at Level 4, regardless of its performance at lower levels.
  4. Project success eventually becomes historic, as the work of the project and/or its major deliverables becomes obsolete. The timeframe during which this occurs will vary with the project discipline (a building may last 100 years or more, but a computer system might be replaced within five years). The historic success of the project remains the highest level achieved during its operational life.
  5. Project “failure” at one or more levels is an indicator of the possible need for organizational learning and development at that level, to avoid repetition in subsequent projects.

The framework comprises generic criteria that should be applicable to most projects from most disciplines. While it offers improvements on extant approaches to defining project success, there is no panacea for differences of opinion arising between individuals on whether a project is successful or not. Success at a particular level may still be a matter of majority opinion. However, consistent with advice from the literature (discussed in the previous section), this problem can be mitigated by each project defining specific performance criteria within each success level (as relevant to the project), in the business case and project plan for later assessment of performance outcomes. If these project-specific performance criteria are defined in a manner that is consistent with the success levels as described in Table 1, meaningful comparison of project outcomes can still be undertaken.

Information Systems Project Case Illustrations

The following information systems examples illustrate project success and failure using the proposed framework.

Sydney Water's CIBS Project – A Failure

As reported in NSW-AG (2003), in June 2000, Sydney Water, a government owned utility company, let contracts to build and implement a Customer Information and Billing System (CIBS). The project was intended to improve service to customers, fill gaps in existing information systems, and provide business efficiencies. The new system was to integrate with 12 existing internal systems and 60 external party interfaces. The system was expected to be operational by February 2002, at a cost of $38.2 million (AUD). Due to the board's concerns about delays, excessive costs (the estimate had blown out to $135.1 million (AUD)), and failure to reach acceptable standards, the project was terminated on 30 October 2002 at a loss of $61 million (AUD). A government audit review of the project found deficiencies in:

  • Project governance by management and the board. Management did not keep the board informed of important project issues and the board did not oversee the project as effectively as it might have, given its significance and complexity. For example, the project was approved before an IT architecture was defined. After an architecture was put in place, CIBS was found to fail 19 of 20 architectural requirements.
  • Project specification, project management, and interface with users. Project planning and specifications were inadequate, resulting in a stream of change requests, and communication with users was poor. An integrated project plan was not maintained throughout the project and the business case was not maintained. Also, testing was poorly managed.
  • Contractual arrangements. Contract administration was deficient. For example, the contract was weaker than the request for tender document. Furthermore, one contract variation transferred significant responsibilities and risk from the contractor back to Sydney Water.
  • Risk management. Risk management was ineffective within the project and at the organizational level. There was a view that outsourcing the project transferred the risk to the contractor.

While this project clearly failed at Level 2 (because it did not complete), the audit report indicates that there were significant failures at Level 1 that contributed to the delays and cost overruns, resulting in loss of confidence in the project and its eventual termination. A number of recommendations for improvement were made for future projects.

St. Henry's Hospital's BMS Project – Success at Level 1 (Almost)

This project also “failed” in that it did not complete, but failure at the process level was less obvious. St. Henry's is a pseudonym for a government-owned hospital that initiated a half-million dollar (AUD) project in 2002 to build a web-based Bed Management System (BMS). The scope of the project was to specify, design, build, and implement an information system to support real-time bed bookings, scheduling, bed status reporting, and bed data publication to key stakeholders (including ambulance services and the emergency departments of other hospitals in the region). It was also to integrate with the hospital's patient administration and clinical systems.

Six months into the project, the steering committee became aware of plans by the organization that administered all government-owned hospitals in that jurisdiction to implement a new patient administration system in each of its 250 hospitals and a centralized electronic health record database system. This created unexpected dependencies for the local project initiative at St. Henry's. The BMS project was halted for a protracted investigation of the options available to St. Henry's.

The jurisdiction-wide system rollout would not provide the functionality St. Henry's wanted for its BMS. However, for contractual reasons (with the system providers), St. Henry's could not proceed with developing extensions to the new systems ahead of completion of the roll-out. Consequently, the BMS project was eventually discontinued (from the hospital's perspective, the project was “cancelled” rather than “failed”).

Ironically, in terms of Level 1 processes, the project was well managed by a contracted project management professional up to the time of its termination. A detailed business case, risk analysis, project plan, functional requirements, architectural design, database design, and interface specifications were developed and maintained, and a hospital process review was conducted to produce an efficient practice model for the new system. Furthermore, the project foresaw the possibility of its own demise. The biggest risk identified for the project was that it might encounter dependencies on other system projects over which it had no control. However, as in surgery, the notion that “the operation was a success but the patient died” carries little comfort for “patient” stakeholders.

More typically, a Level 1 success might be determined at project closeout for a project that failed one or more of the Level 2 measures (time, cost, or project scope) for reasons other than process failure.

CompuSys' CONFIG System Project – Success at Level 2

Markus and Keil (1994) recounted a classic case of a textbook system development project that produced a system that users would not use. CompuSys, a computer company, built CONFIG, an expert system, to check the configuration specifications of new computer systems, prepared by sales representatives, before the sale. This would avoid losses from supplying necessary components that were omitted from the quotation. The system was a technical success in terms of Levels 1 and 2 of the framework, and produced better configurations than the average sales representative, but the sales force would not use the system. That is, it failed at Level 3.

Having determined that the problem must have been human factors, the system developers embarked on a major redevelopment to improve the user interface. This time, the users agreed that the second version of CONFIG was far superior to the first but they still would not use it. On investigation, it was found that sales reps had no motivation to use the system because they were measured on sales, not occasional losses due to missing cables. Also, the system made it harder rather than easier to generate quotations because CONFIG was not integrated with PQS, the Product-pricing and Quotations-generation System. Markus and Keil (1994) concluded that this outcome was the result of “process suboptimization through subprocess optimization.” That is, a myopic focus on the configuration process blinded CompuSys to the context of the larger sales quotation process.

Admin's GovNet Project – Success at Level 3

In another example of technical myopia, in the second half of the 1990s, Admin (a pseudonym for a government department) responded to a new whole-of-government policy to extend transaction services to the public via a web-based application I will call GovNet. The services were provided by interfacing new web applications to Admin's existing in-house online transaction processing system. The system presented several technical challenges because web application development platforms were evolving and still immature at that time, and the main system had data management controls that the add-on applications could easily circumvent if great care was not taken. Despite these challenges, the project was completed successfully, gaining strong acceptance and use by early adopters of web-based services.

It quickly became apparent, however, that Admin had not thought through the operational implications of such a relatively easy extension to its transaction system (technical challenges aside). Naturally, users expected to be able to access the systems on the Internet 24 x 7, but the base transaction system, which included the database, was designed to operate only during normal business hours. Out-of-hours time was dedicated to batch updates, batch printing and reporting, extracts for dependent systems, backups, and other system “housekeeping.” Also, Admin had insufficient staff and no budget to run multiple help desk shifts to provide telephone support to online users at night. At best, availability of the web applications could only be extended a few hours either side of the normal business day, and without help desk support. This resulted in considerable embarrassment for the department and great pressure to resolve the problem, which could not be done in any meaningful practical way within the design of the existing base system. Despite achieving the immediate goal of compliance with government policy, the net benefits of the project to the organization at Level 4 were negative. Consequently, the project's success was capped at Level 3 (given acceptance by users of the limited system availability).

DMV's LARS Project – Success at Level 4

In 1990, an Australian DMV (Department of Motor Vehicles) initiated a project to develop and implement a new Licensing And Registration System (LARS) to replace its legacy batch processing mainframe system. The system was to be completed in 18 months at a cost of $29 million (AUD). The system was to be developed in-house using contract developers and new, high-productivity development technologies that would automatically generate programs from design specifications. The project was highly innovative on several fronts. The system design was innovative and required integration of new technologies that had been previously untried. The development method was also new and unproven for the chosen system platform. Furthermore, the project was well organized into five teams for development (the largest), systems environment, data conversion, change management, and procedures.

From the beginning, major problems confronted each team. Delays ensued, compromises had to be made, due dates were frequently extended, and eventually the implementation had to be split into two phases with scaled-down deliverables to get a system implemented. A licensing module was implemented nine months late, followed by the registration component a year later. On implementation, as the LARS general manager expressed it, “All hell broke loose.” The system was slow and regularly locked up, customer data records were inaccurate or could not be found at all, standard policies were enforced inappropriately, and customer queues became intolerably long at the DMV office counters. Public outcry ensued. However, in the following months and years, the problems were progressively and fully resolved. Some estimates put the final cost of LARS at $80 million (AUD).

Curiously, while the project struggled with technical and user problems, it did achieve the major business objectives set for it. It replaced an unsupportable legacy system and saved the organization $20 million (AUD) per year in costs associated with the old way of administering driver licenses and vehicle registrations. Indeed, it was considered such a business level success that the Department won an industry Award for Excellence for the new system. The project succeeded at Level 4 despite failing against the criteria of all lower levels.

American Airlines' SABRE System Project – Success at Level 5

A final example is another classic case from the IS literature. American Airlines is widely recognized for developing SABRE, the world's leading computerized airline reservation system. The development of SABRE is well chronicled (Copeland & McKenney, 1988; Copeland, Mason, & McKenney, 1995). Through most of its history, SABRE dominated the US airline industry, successively evolving from a computer-based airline reservations system to a passenger service system, a sales distribution system, and an electronic travel supermarket, setting the benchmark for other airlines to adopt or compete against. Indeed, it is reputed that SABRE was so successful that the system became more profitable for American Airlines than the airline business itself.

Discussion

This paper aims to contribute to the formalization of project management as a theoretical discipline by synthesizing extant research into a multilevel framework that offers the potential to unite disparate perspectives on projects in a consistent and uniform way of determining outcomes. Reaching consensus on defining project success is likely to smooth the path to a better understanding of the responsibilities and boundaries of project management as an enabler of change within organizations and the wider stakeholder community. While consensus is not necessary for theoretical progress (indeed, the reverse is usually true), a reference framework can aid in providing clarity of focus by mapping the high level footprint of the discipline at a particular stage of its development, and depth of focus by highlighting key dimensions of projects that stakeholders view as being important to them. Much of the current focus of project management research and practice is limited to prescriptions for achieving success at Level 2, yet project stakeholders indicate broader interests in projects as a vehicle for delivering change and new capabilities.

The paper has limitations. The proposal is conceptual, developed from a review of the literature and illustrated empirically with six case examples. The relevance and completeness of the criteria in the framework and the framework's practical utility must be tested by application to many projects over time. Also, generalization of the framework to apply to other project disciplines requires validation in practice. Ultimately, as with projects, it is likely to be project stakeholders who determine the value of the framework.

The paper has implications for research and practice. The paper extends the current body of knowledge on project management by providing a way to integrate current thinking on the definition of project success. It also extends this thinking by explicating two additional project success criteria. This integration points to a broader scope of relevance of projects than that defined by the traditional project life cycle. The perceptions of stakeholders suggest that they value outcomes attributable to the original project well beyond closeout. Consequently, future research has the opportunity to investigate the role of projects and project management in satisfying stakeholder expectations and perceptions beyond the narrow confines of “on time, within budget, and to specification” performance.

For practice, the framework enables different project interests, perspectives and outcomes to be normalized in parallel rather than in competition with each other. It uncouples project management success (Levels 1 and 2) from the post-project benefits that subsequently flow to stakeholders (Levels 3 to 5), enabling acknowledgement of the highest value derived, regardless of tactical performance. A practical danger here, however, is that success at a high level is not misinterpreted as excellence at all subordinate levels. Each level has meaning only within that level. “Failure” at a lower level may be equally, if not more, important than the success if that failure points to a need for improvement. For example, that the DMV's LARS system ultimately “succeeded” in achieving the required business benefits may obscure the failures at Levels 1 to 3 for this project (even though the net financial benefits were diminished by up to $50 million (AUD) in additional development costs). The critical importance of those failures comes into relief for the next major development project. The outcome of each successive project is zero-based, save for the experiential learning in managing projects that the organization accumulates over time.

Conclusion

This paper has argued that project management as a formal discipline would benefit from a unifying representation of project success. Such a contribution would help focus research attention on aspects of projects and project management that are important to stakeholders. It would also facilitate communication and comparison through a common language of project success. Acknowledging the realities of practice, this reference framework would have to accommodate the multiple expectations and perceptions that arise from different stakeholder perspectives at different positions of interest and time. The multilevel framework of project success developed in this paper aims to fulfill these objectives.

Organizations in all disciplines are increasingly finding that project-based work, supported by project management best practice, offer considerable utility over traditional functional designs in managing discrete activities. As the discipline of project management matures, this trend is likely to continue. Therefore, it is important that the discipline continues to develop and periodically formalize its position (in agile software development, this is called refactoring). The challenge of this paper for practice is to recognize and celebrate success in a way that is meaningful to others, aided by the proposed framework, while progressively improving project management capabilities. The challenge for research is how to respond to the wider community of project stakeholders whose interests lie beyond the project life cycle as currently conceived.

References

Atkinson, R. (1999). Project management: Cost, time and quality, two best guesses and a phenomenon, its time to accept other success criteria. International Journal of Project Management, 17(6), 337-342.

Baccarini, D. (1999). The Logical Framework Method for defining project success. Project Management Journal, 30(4), 25-32.

Baker, B. N., Murphy, D. C., & Fisher, D. (1988). Factors affecting project success. In D. I. Cleland & W. R. King (Eds.), Project management handbook (2nd ed.), pp. 902-919. New York: Van Nostrand Reinhold.

Ballantine, J., Bonner, M., Levy, M., Martin, A., Munro, I., & Powell, P. L. (1996). The 3-D model of information systems success: The search for the dependent variable continues. Information Resources Management Journal, 9(4), 5-13.

Chrissis, M. B., Konrad, M., & Shrum, S. (2007). CMMI: Guidelines for process integration and product improvement (2nd ed.). Boston: Addison-Wesley.

Ciborra, C. & Jelassi, T. (Eds.). (1994). Strategic information systems: A European perspective. Chichester, UK: John Wiley & Sons.

Cleland, D. I. & Ireland, L. R. (2007). Project management: Strategic design and implementation (5th ed.). New York: McGraw-Hill.

Cooke-Davies, T. (2002). The “real” success factors on projects. International Journal of Project Management, 20(3), 185-190.

Cooke-Davies, T. (2004). Project success. In P. W. G. Morris & J. K. Pinto (Eds.). The Wiley guide to managing projects, pp. 99-122. Hoboken, NJ: John Wiley & Sons.

Copeland, D. G. & McKenney, J. L. (1988). Airline reservations systems: Lessons from history. MIS Quarterly, 12(3), 353-370.

Copeland, D. G., Mason, R. O., & McKenney, J. L. (1995). Sabre: The development of information-based competence and execution of information-based competition. IEEE Annals of the History of Computing, 17(3), 30-56.

DeLone, W. H. & McLean, E. R. (1992). Information systems success: The quest for the dependent variable. Information Systems Research, 3(1), 60-95.

DeLone, W. H. & McLean, E. R. (2003). The DeLone and McLean model of information systems success: A ten-year update. Journal of Management Information Systems, 19(4), 9-30.

De Wit, A. (1988). Measurement of project success. International Journal of Project Management, 6(3), 164-170.

Emery, J. C. (1990). Misconceptions about strategic information systems. MIS Quarterly, 14(2), vii-viii.

Gaddis, P. O. (1959). The project manager. Harvard Business Review, 37(3), 89-97.

Jugdev, K. & Müller, R. (2005). A retrospective look at our evolving understanding of project success. Project Management Journal, 36(4), 19-31.

Kerzner, H. (2003). Project management: A systems approach to planning, scheduling, and controlling (8th ed.). Hoboken, NJ: John Wiley & Sons.

Kloppenborg, T. J. & Opfer W. A. (2002). The current state of project management research: Trends, interpretations, and prediction. Project Management Journal, 33(2), 5-18.

Koskela, L. & Howell, G. (2002). The underlying theory of project management is obsolete. Proceedings of the PMI Research Conference, 293-302.

Lyytinen, K. & Hirschheim, R. (1987). Information systems failures: A survey and classification of the empirical literature. Oxford Surveys in Information Technology, 4, 257-309.

Markus, M. L. & Keil, M. (1994). If we build it they will come: Designing information systems that users want to use. Sloan Management Review, 35(4), 11-25.

Morris, P. W. G. (1994). The management of projects. London: Thomas Telford.

Morris, P. W. G. (2002). Science, objective knowledge, and the theory of project management. Proceedings of the Institution of Civil Engineers – Civil Engineering, 150(2), 82-90.

Nelson, R. R. (2005). Project retrospectives: Evaluating project success, failure, and everything in between. MIS Quarterly Executive, 4(3), 361-371.

NSW-AG, (2003). Review of Sydney Water's Customer Information Billing System. NSW Auditor General's Report to Parliament, 1. Retrieved June 2005 from http://www.audit.nsw.gov.au/agrep03v1/SpecialRevSydneyWaterCIBS.pdf

Pinto, J. K. (2004). The elements of project success. In D. I. Cleland (Ed.), Field guide to project management (2nd ed.), pp. 14-27. Hokoben, NJ: John Wiley & Sons.

Pinto, J. K. & Covin, J. C. (1989). Critical factors in project implementation: A comparison of construction and R&D projects. Technovation, 9(1), 49-62.

Pinto, J. K. & Mantel Jr., S. J. (1990). The causes of project failure. IEEE Transactions on Engineering Management, 37(4), 269-276.

Pinto, J. K. & Slevin, D. P. (1988a). Project success: Definitions and measurement techniques. Project Management Journal, 19(1), 67-72.

Pinto, J. K. & Slevin, D. P. (1988b). Critical success factors across the project life cycle. Project Management Journal, 19(3), 67-74.

Porter, M. E. & Millar, V. E. (1985). How information gives you competitive advantage. Harvard Business Review, 63(4), 149-160.

Sauer, C. & Reich, B. H. (2007). What do we want from a theory of project management? A response to Rodney Turner. International Journal of Project Management, 25(1), 1-2.

Sauer, C., Gemino, A., & Reich, B. H. (2007). The impact of size and volatility on IT project performance. Communications of the ACM, 50(11), 79-84.

Schultz, R. L., Slevin, D. P., & Pinto, J. K. (1987). Strategy and tactics in a process model of project implementation. Interfaces, 17(3), 34-46.

Seddon, P. B. (1997). A respecification and extension of the Delone and McLean model of IS success. Information Systems Research, 8(3), 240-253.

Seddon, P. B., Staples, S., Patnayakuni, R., & Bowtell, M. (1999). Dimensions of information systems success. Communications of the AIS, 2, 2-60.

Shenhar, A. J. (2001). One size does not fit all projects: Exploring classical contingency domains. Management Science, 47(3), 394-414.

Shenhar, A. J., Dvir, D., & Levy, O. (1997). Mapping the dimensions of project success. Project Management Journal, 28(2), 5-13.

Shenhar, A. J., Dvir, D., Levy, O., & Maltz, A.C. (2001). Project success: A multidimensional strategic concept. Long Range Planning, 34(6), 699-725.

Shenhar, A. J., Poli, M., & Lechler, T. (2000). A new framework for strategic project management. In T. Khalil (Ed.), Management of technology VIII. University of Miami, Miami, FL.

Shenhar, A. J., Tishler, A., Dvir, D., Lipovetsky, S., & Lechler, T. (2002). Refining the search for project success factors: A multivariate, typological approach. R&D Management, 32(2), 111-126.

Shenhar, A. J. & Wideman, R. M. (1996). Project management: From genesis to content to classification. INFORMS Conference, Washington DC, May, downloaded from http://www.maxwideman.com/papers/genesis/genesis.pdf

Slevin, D. P. & Pinto, J. K. (1986). The project implementation profile: New tool for project managers. Project Management Journal, 17(4), 57-70.

Söderlund, J. (2004). Building theories of project management: Past research, questions for the future. International Journal of Project Management, 22(3), 183-191.

Turner, J. R. (2004). Five necessary conditions for project success, International Journal of Project Management. 22(5), 349-350.

Wateridge, J. (1998). How can IS/IT projects be measured for success? International Journal of Project Management, 16(1), 59-63.

Yourdon, E. (1997). Death march: The complete software developer's guide to surviving “mission impossible” projects. Upper Saddle River, NJ: Prentice Hall PTR.

_____________________________

1 I use contingency variables in the sense of organizational contingency theory, such as discussed by Shenhar (2001), not in the common sense, in the context of project planning, of provisioning for uncertainty. Classical contingency theory argues that project performance is contingent upon congruence between a project's structure and its environment. That is, a project must be designed and managed according to its contextual circumstance. The variables identified by this research stream point to contexts in which different project designs and management actions may be necessary for the project to succeed.

This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI or any listed author.

© 2008 Project Management Institute

Advertisement

Advertisement

Related Content

Advertisement

Publishing or acceptance of an advertisement is neither a guarantee nor endorsement of the advertiser's product or service. View advertising policy.