Twenty-first century project success measures

Abstract

Project success as an object of study has previously been characterized as inclusive, ambiguous, and multidimensional. It is the intent of this research to further investigate how project management professionals define project success factors by answering the following questions: (1) Are previously identified success measures the measures current practitioners use? (2) Are some of the success measures more important than others? (3) How do these success measures vary by industry or type of project? These project success measures were derived from the literature and empirically analyzed in previous studies. These are the project success measures identified in this paper: meeting agreements, the firm's future, customer success, and project teams. The success measures that practitioners are currently using are both tactical and strategic. The objective of this research is to interpret which success measures are of most importance to practitioners and to better understand why those factors may fluctuate, depending on the stage of the project.

Keywords: Project management; project life cycle; project success factors

Twenty-First Century Project Success Measures: Evolution, Interpretation, Direction

During the last three decades, academics and practitioners have expanded the triple constraint measures of cost, schedule, and performance as indicators of a project's success to include measures associated with customer satisfaction, teams, and the future of the firm. Much of this research is survey based and this empirical analysis involves opinions of both project management practitioners and researchers.

Research considering the role of the executive sponsor in project success has culminated in a stage-by-stage empirical identification of executive sponsor behaviors that contribute to project success. Remarkably, the same project success factors emerged in all four studies. This study undertakes a rich interpretation of these project success factors by project management professionals. The current research aims to answer the following questions: (1) Are previously identified success measures the measures current practitioners use? (2) Are some of the success measures more important than others? (3) How do these success measures vary by type of project and industry?

The paper is organized in the following manner: The first section outlines a summary of project success research in the 21st century, detailing the expansion of the original triple constraint notion. Contributions to project success from the authors' sponsor-based empirical analyses form the framework for the research methodology. Next, the research methodology is presented, detailing project management professionals' examination of project success factors by stage. The paper concludes with an interpretation of results and implications for further investigations of project success behaviors.

Background

Project Life Cycle Context

In investigating project success measures, a project life cycle context facilitates an understanding of how projects vary from one point in time to another. Project life cycle models ensure a consistency of approach to management processes, provide a communication vehicle, make pre-implementation activities (i.e., cost benefit/analysis) happen, and provide for quality assurance (Bryde, 2007). The concept of a project life cycle is that during a given stage certain requirements must be met or the project is not allowed to pass to the next stage. Despite the unique characteristics projects from different industries encounter, many people use a generic project life cycle model. The most basic model often includes initiation, planning, executing, and closing the project (Kloppenborg, 2012; Project Management Institute, 2008).

Project Initiating

The initiating stage begins with the first idea of a potential project. This may include selecting which projects the firm chooses to undertake. Once it is determined that the project in question should be performed, the initiating stage typically consists of writing and approving a project charter. The charter makes the project official and grants the project manager authority to proceed into the planning stage (Project Management Institute, 2008, pp 44–45).

Project Planning

The planning stage begins when the project charter has been accepted and concludes with the acceptance of the entire project plan. The project plan includes all detailed planning documents that are required such as schedule, budget, scope, risk, quality, and communications. The start of this stage is generally easily defined, but on many projects, some work is planned and then implemented in an overlapping or iterative fashion. The distinction at the end of the planning stage is not always easily understood and can be misunderstood by members of the project team (Project Management Institute, 2008, pp 46–55). For this research, respondents were instructed to consider planning activities—whether all planning concluded before execution started or not.

Project Executing

The executing stage of the project life cycle begins with completion of project plans (or at least approval of partial plans so the project can start). The execution stage ends with an unambiguous acceptance of project deliverables. Regardless of industry, most project managers agree that demonstration of the results that are accepted by the customer constitutes the end of the executing stage. (Project Management Institute, 2008, pp 55–64)

Project Closing

Just as the project executing stage concludes, the project closing stage begins with acceptance of the project deliverables. When all project work is completed, including reassignment of personnel, closing of books, capturing of lessons learned and verification of benefits, the project closing stage is considered complete (Project Management Institute, 2008, pp 64–65).

Project Success

Historically, project success has been described by the “iron triangle” of cost, schedule, and performance. In this “operational mindset,” metrics can often be incomplete and misleading. While it is still important to achieve performance objectives subject to constraints of time and money, many researchers have expanded the previously widely accepted notion of project success.

Shenhar, Dvir, Levy, and Maltz (2001) demonstrated empirically that project success is composed of the four factors of efficiency (cost and schedule), customer impact (multiple measures), business success (commercial success and market share), and future preparation (multiple measures). Shenhar et al.'s study also suggested that various measures of success could be measured at different points in the project life cycle.

Pinto (2004) suggested measuring 13 performance measures that ultimately, he postulates, cluster into four factors of project efficiency, impact on customer, business success, and future potential.

Jugdev and Müller (2005) summarized project success research by first considering tactical issues starting with the iron triangle and increasingly including more strategic and customer-oriented measures. The unambiguous results from their research indicate that success needs to be envisioned in both efficiency (operational) and effectiveness (strategic) terms. Research streams used to explain project success include project management maturity models, balanced scorecards, and success factors. Increasingly, project success has been seen as multidimensional and potentially measured differently at different points in the project life cycle.

In no uncertain terms, executives view satisfying the customer as the most important measure of project success (Dilts & Pence, 2006). In a mixed-method study, Dilts and Pence used recurring business, customer satisfaction, end-user satisfaction, team satisfaction, stakeholder satisfaction, and supplier satisfaction as the most critical project success criteria. In their research, rates of success did not vary by project type or industry. Shenhar and Dvir (2007) expand upon Pinto (2004) and Shenhar et al. (2001) to include impact on the project team as an equally important project success factor. In total, the instrument uses 28 items to measure project success. Müller and Turner (2007) concluded that team and end-user satisfaction are the most important factors.

Scott-Young and Samson (2008) investigated capital projects in the process industries. They unpack project success criteria, discovering that different project methods and behaviors correlate with different project success measures. Their recommendations include using larger sample sizes to enable simultaneous path analysis of all outcomes and inclusion of a broader range of project success criteria to include client/end-user satisfaction and project team satisfaction. Bryde (2008) used an 11-item project success measure, including both shorter-term objective measures such as time, cost, and specifications with longer-term perceptual measures. Geoghegan and Dulewicz (2008) used a 12-item project success measure that categorizes the behaviors into the three factors of project delivery, usability of project outputs, and value of project outcome to users. Khang and Moe (2008) considered what needed to be accomplished at each life cycle stage and determined project success according to these five items: impact on beneficiaries, institutional capacity, project reputation, extension of project work, and sustainability of outcomes. What began to emerge in 2008 is a more empirical approach that considers multiple project success measures; an attention to the project life cycle; and an emphasis on considering the behaviors that correlate with specific outcome measures.

Ika (2009) summarized trends in published findings of project success. The study tracks the historical development starting with the “iron triangle” and ending in the 21st century, with the added success criteria of client/end user success, benefits to stakeholders, and benefits to project personnel. Project success as an object of study is described as inclusive, ambiguous, and multidimensional.

Malach-Pines, Dvir, and Sadeh (2009) and Kloppenborg, Manolis, and Tesch (2009) tested success measures advanced by Pinto (2004) and Shenhar et al. (2001), which revealed the measures grouped into the same three factors of meeting agreements, impact on the customer, and future of the firm. One key difference, however, is that the more current studies found adherence to technical specifications factored with cost and schedule, not with customer items as the earlier studies showed.

Devine, Kloppenborg, and O'Clock (2010) suggested using a balanced scorecard approach to measuring project success would yield more accurate results. This approach states that no perspective is more important than the needs of the customer and suggests that success indicators should be considered at each stage of the project life cycle. Shao, Müller, and Turner. (2010) grouped project success into the six categories of business success, stakeholder satisfaction, preparation for the future, efficiency, social effects, and team. Papke-Shields, Beise, and Quan (2010) used six measures for project success: cost, time, technical specifications, quality requirements, client satisfaction, and business objectives.

Al-Tmeemy, Abdul-Rahman, and Harum (2011) surveyed managers of building projects in Malaysia and found their responses fell into three success factors of project management, product, and market. Kloppenborg, Tesch, and Manolis (2011) surveyed both managers and executives from a wide variety of industries and locations (primarily North American) regarding behaviors of sponsors during project planning. They also identified three success factors: agreements, customer, and future. Tesch, Kloppenborg, and Manolis (2011) added questions regarding the project team as suggested by Shenhar and Dvir (2007) in their study of sponsor behaviors during the project executing stage. They discovered four success factors of agreements, customer, future, and teams.

A chronological summary of major studies in project success in the 21st century appears in Table 1. The studies are marked by type as conceptual (C), empirical (E), summary (S), or mixed-method (M). Where a specific number of individual items were tested or postulated, that number is shown and the specific outcomes factors are reported.

Table 1: Major 21st Century Project Success Measure Studies.

Method

In this research, we further investigated established project success factors by stage in executive sessions with project management professionals in order to answer the following questions: (1) Are previously identified success measures the measures current practitioners use? (2) Are some of the success measures more important than others? (3) How do these success measures vary by type of project and industry? These project success measures were derived from the literature and empirically analyzed in previous studies. Project success measures identified are meeting agreements, the firms' future, customer success, and project teams.

Table 2 presents the project success factors and items associated with each identified factor.

Sample, Procedures, and Measures

In a series of studies examining successful project sponsor behaviors in the initiating, planning, executing, and closing stages, 677 respondents were asked on four separate occasions to indicate the extent to which a number of performance dimensions is significant in terms of a project's success (1 = No extent; 7 = great extent). Each study started with a literature review, then proceeded through one or more focus groups with experienced project managers and executives, and ended with an email survey. No person participated in any two consecutive stages of the research and no person who was in a focus group was asked to answer the survey for the same project stage. Table 3 presents summary demographic data across studies. Note the wide variety of respondents along demographic dimensions such as industries and functional areas. The respondents are very experienced, with about half having at least 25 years of experience and quite a few are PMP^® credential holders.

In a recent executive session, 33 project management practitioners (as shown in the final column of Table 3) were asked to consider the proposed research questions. In the executive session, the researchers presented the findings from each of the four stage-specific surveys. The participants were divided into four groups, so each participant could contribute his or her ideas in a small group on each project stage. The questions for each stage were asked by a faculty member and the responses were captured by two students with laptops. The participants were asked the following questions:

• Are these the success measures they use?
• Are some of the measures more important to them than others?
• Does the use of certain success measures vary by industry or project type?
• Does satisfying one measure lead to satisfying others? and
• Does satisfying one measure lead to worse results in others?

Table 3: Summary Demographic Data Across Studies.

Results

Table 4 presents the factor correlations between each of the significant success factors/dimensions according to each stage of the project life cycle. The team's factor was added to the research in the executing stage analysis based upon the suggestion of Shenhar and Dvir (2007). The reported Cronbach's alpha (0.47) indicates a poor internal consistency among factor items. The highest correlation between teams and other success factors in the executing stage was 0.47 between the teams and customer success factors. Both customer success and teams have an item regarding satisfaction and perhaps that is why they are the most highly correlated factors. Due to both the poor internal consistency and high correlation with customer success, the team factor was not used in the closing stage. Note also, by the gray areas, that meeting agreements did not apply during the initiating stage, and customer success did not apply during the closing stage. In both cases that was due to how each stage was defined. The initiating stage is defined to end when the charter is approved, and only after that are agreements in terms of firm commitments to cost, schedule, and technical performance established. The closing stage is defined to begin after the customer formally accepts the project deliverables.

The highest correlation with the meeting agreements factor is found in the executing and closing stages, where meeting agreements is correlated with the firm's future at .37 and .36, respectively. The highest correlation with the customer success factor is .42, in the initiating stage with the firm's future.

Table 4: Factor correlations and Cronbach alphas*.

* Cronbach alpha estimates are bolded on the diagonal of each correlation matrix.

** Gray areas indicate that these factors were not significant in these stages.

Research question two considers whether some of the success measures are more important than others. To address the relative importance of project success measures within and between each life cycle stage, a series of one-way, repeated measure ANOVA models (one model each according to the life cycle stage study) with pairwise comparisons using the Tukey-Kramer test to compare success factors within each project life cycle stage. Table 5 presents the results of these analyses. The team's factor was omitted from this analysis based on previously established poor internal reliability.

In the initiating stage, we find that pleasing the customer (i.e., customer success) is significantly more important than a firm's future (see Table 5). In the planning stage, we find that meeting agreements and customer success are both significantly more important than a firm's future. Further, we find that customer success is more important than meeting agreements (see Table 5). Interestingly, analysis of the executing stage data revealed the exact same pattern of results as were found for the planning stage (see Table 5). That is, customer success was the most important outcome variable, followed by meeting agreements and a firm's future, respectively. In the closing stage, results reveal that meeting agreements is significantly more important than a firm's future.

Table 5: Project success measures: Means and mean differences within each project life cycle stage.

Note. C = Customer, A = Agreements, F = Firm's Future

To answer the second part of the first research question (Does the relative importance of project success measures vary across life cycle stage?), we again conducted a series of one-way ANOVA models, followed by pairwise comparisons using the Tukey-Kramer test. Results reveal one significant finding: customer success is more important in the initiating stage versus the planning stage.

Discussion

The first research question is: What measures are actually used to measure project success? The participants in our executive session expressed mixed opinions. Many agreed with all four of the factors (meeting agreements, customer success, teams, and firm's future). Many looked at individual items within some of the factors and stated that they used some, but not all of the individual factors.

Two enhancements to our initial model were suggested. First, teams are an important success measure. It is highly advantageous for a company to have more capable and more enthusiastic team members. In fact, several executives stated that team success is one of the factors that lead to improvement in the firm's future. A second enhancement is to include customer success during project closing. Even though the customer has already accepted the deliverables, it does not mean they no longer need help. The executives in our session reminded us that if project managers merely take a transaction approach to projects, they are likely to be somewhat successful. However, if they take a relationship approach and work hard to ensure that the customer can profitably use the results of the project, they are likely to have customers who are more loyal and have more follow-on work.

The second research question is: Are some success measures more important than others? The clear answer is yes—customer success is the most important. Table 5 shows the customer is significantly more important at each stage it was considered. Our respondents loudly echoed that sentiment with comments such as, “without customers there would be no need for projects.” They pointed out that in fast-paced environments, specifications, budgets, and schedules often change, but the constant is that some customer needs the project deliverables.

Table 5 also shows that meeting agreements is more important than the firm's future in each stage where it is considered. What our executives also told us is that teams are more important (even if we apparently did not have the best measures) than the firm's future. They said that in order to enhance the firm's future, one needs to continually ensure that the project is aligned with organizational goals, and to ensure each of the other success measures (customer, agreements, and teams) is achieved.

Our third research question is: Do success measures vary by type or project, industry, or other factors? We were not able to answer that question statistically due to the wide variety of respondents (see Table 3). Although one of the strengths of this study is the wide variety of respondents, the sample sizes within each group did not allow for direct comparison. The executives attending our interpretation session, on the other hand, had strong opinions on differences. These are some of the suggestions:

• In consumer goods, a project's market share is most important;
• In food industry projects, the firm's future is most important;
• In healthcare projects, customer satisfaction is most important;
• In some industries, safety or adherence to government regulations is the most important;
• The most important factor depends on the organization's strategy, for example, a first mover strategy would dictate schedule as most important; and
• Other things such as the company size, length of project life cycle, or organizational culture may dictate the most important criteria.

Figure 1 illustrates previously identified project success factors in terms of their current versus future orientation, effectiveness versus efficiency objectives, and in terms of their consideration as more tactical versus strategic. Based both upon our studies and those of many others, we believe each of these is a useful consideration when measuring project success. Meeting agreements (essentially the old “iron triangle”) is rather efficiency and present oriented, putting it in the tactical corner. This is still important. However, an emphasis on customer success reminds us why a project was undertaken in the first place. Customer success is an effectiveness measure with both current and future time orientations. Likewise, a consideration for project teams is largely an effectiveness view with both current and future time orientations. The firm's future obviously is a future orientation and it can include both efficiency and effectiveness measures.

Suggestions for Future Research

Our samples represent diverse functional areas, industries, and project types, and, as such, allow us to ascertain some degree of generalizability for both our measures and findings regarding project success. At the same time, however, this diversity precludes us from making valid statistical comparisons across these different functional areas, industries, and project types as the sample sizes comprising each specific group—whether according to functional area, industry, or product type—are too small. In addition, the vast majority of subjects comprising the four samples resided in North America. Thus, future research should test and compare our variables and the relationships between them using sufficiently large sets of data collected from specific and diverse groups of project management personnel according to functional areas, industries, product types, and/or countries and cultures.

The individual items we used in our studies originally came from important the work of Shenhar et al. (2001), Pinto (2004), and others. However, as can be seen in Table 1, many other researchers have contributed measures—some quite similar and some a bit different. In the interest of response rate, we attempted to keep our list of measures fairly short—never more than 17 individual items. Other researchers could look carefully at the items proposed in a wider variety of those studies in hopes of fine-tuning the measures. Although most of our factors had high Cronbach alphas, the team factor did not. Our instincts and our executive participants tell us there are important measures for project teams. Perhaps other researchers can experiment with team measures.

The studies we used dealt with behaviors of project sponsors and how they related to project success. Future studies that deal with behaviors of project managers or other inputs and comparing them with project success may yield new nuances on how to interpret project success. Finally, we believe most project management research should answer both the questions of what people actually do and how those behaviors relate to a contemporary approach to project success. The old “iron triangle” is still valid, but it is no longer enough. At the very least, any project management research should also consider the impact of whatever is being studied upon the project customers.