What's the benefit?
Challenges in demonstrating the value of project management
Investments in project management (PM) compete with other potential organizational investments. As with any investment, organizations need to be assured of a concrete return. Without the ability to clearly define its value and impacts, PM joins the long line of other functions or initiatives (i.e., TQM, Information Systems, Training, Human Resources) struggling to prove their worth to organizations.
Although the holy grail of demonstrable PM value is often discussed, and even proclaimed, in consulting and practitioner literature, the actual value resulting from investments in PM has been hard to define let alone measure. Few rigorous studies have been undertaken and those that exist struggle to provide indisputable and strong evidence. Where quantitative evidence exists (see Kwak & Ibbs, 2000; Cooke-Davies, 2002a; Reginato & Ibbs, 2002; Ibbs, Reginato, & Kwak, 2004; Mullaly, 2004), it is tantalizing but fragmented and incomplete.
PMI has sponsored the launch of a major research project designed to quantify the value of PM. To achieve success, this project must: provide unequivocal and compelling evidence of the value organizations recognize when PM is appropriately implemented; develop practical guidelines for practitioners, consultants and executives to measure the value of PM initiatives; convince the academic, executive, and practitioner communities of the value, rigor and appropriateness of the value statements and practice guidelines.
Learning from Previous Value Research
Proponents of evaluating and measuring the ROI of organizational initiatives state that measuring return on investment is not a new managerial concept, nor is it a particularly complex one. Ultimately it is simply a method of comparing the costs of an initiative with its benefits. The ROI methodology shares the following five steps with cost benefit analysis methods (Phillips & Philips, 2004):
- Identify benefits (including reaction, satisfaction and planned outcomes; learning; application and implementation; business impact; return on investment);
- Convert benefits to monetary value;
- Tabulate the fully loaded costs;
- Identify the intangible benefits;
- Compare the monetary benefits to the costs.
They state that the biggest challenges of conducting this type of evaluation are: ensuring credible, valid results in a reasonable time frame; demonstrating that the results exhibited are appropriately apportioned and attributable to the program or initiative being investigated; and isolating the impacts of the program or initiative from other context-specific and situational factors. The underlying message is that this is not terribly difficult thing to do.
However, project management is not the only organizational initiative that has struggled with equivocal results in ROI research. Information systems (Barua, Lee &Whinston 1996; Barua &Mukhopadhyay, 2000; Powell &Dent-Micallef, 1997) and Expert Systems (Mottiwalla &Fairfield–Sonn, 1998) researchers blame these unsatisfactory results on the absence of effective means of taking into account contextual factors and adequately measuring all impacts. In addition, Soh and Marcus (1995) suggest that all component parts of the IT investment process must be taken into account to get a more effective evaluation of IT benefits. Like the TQM researchers cited above, these researchers all point to the need for a comprehensive measurement framework considering all dimensions of “white-collar” performance (Tuttle &Romankowski, 1985) including: efficiency, effectiveness, productivity, quality, quality of work life, innovation, and productivity. Additional benefits such as autonomy, control and satisfaction could be added to this list.
Empirical research that has tried to evaluate the quantifiable value of organizational initiatives like project management confirm the difficulty in addressing these challenges.
The work of Crosby (1979) provides some illustration of how the cost of quality provides a possible approach to quantifying value. This work found that well developed quality systems had a cost of quality of as little as two percent, where the costs of quality for a poor or neglected quality system could be as high as 20 percent of overall costs. Extrapolating to the project management domain suggests that the impacts of prevention activities (planning, risk management, training, supplier qualification and partnering), appraisal and assessment activities (project tracking, control and communications) and failure-related activities (re-work, claims, loss of future business) will contribute to our understanding of project management value. However, to provide a comprehensive picture of organizational value, researchers suggest that studies of this kind must address three sets of questions (Hackman & Wageman, 1995; Montes, Jover, & Fernandez, 2003) to be successful.
- First, what has actually been implemented? Empirical demonstration that project management has actually been implemented within an organization, and identification of the component capabilities that define project management in each context in terms of function, purpose and practices, is important. Distinguishing between rhetoric and reality will be the challenge (Zbaracki, 1998). Looking at both the nature and the quality of the implementation will be important Going further, the researcher must attempt to tease out which benefits are associated with which portions of PM implementation. That is, if a PMO is implemented without appropriate training (or vice versa) are benefits realized? Exploring the value of either of these separately does not answer the question.
- Second, has this implementation resulted in the improvements in organizational functioning that we would expect? Here we need to assess process criteria effectiveness. Many would expect effective PM to improve the way projects are managed, leading to more satisfied customers and employees/team members and reductions in overruns and delays. What evidence exists to support or refute this?
- Finally, to what degree has this implementation resulted in improvements to bottom line organizational effectiveness in terms of concrete outcomes? Management literature makes it clear that it is very difficult to link the success of any organizational intervention directly to bottom line profit (Eriksson & Hansson, 2003; Kwak & Ibbs, 2000). In these studies, noise caused by the organizational and business context within which the initiative is embedded has been difficult to minimize. Careful attention therefore needs to be paid to this context.
Clearly, empirical evaluation of the value of implementing PM presents significant challenges to researchers, as what must be undertaken to satisfy these 3 different dimensions of assessment involves very different methods and analytic strategies. Ultimately, while it may be tempting to focus directly on the relationship between the entire PM intervention package and global organizational outcomes, cautions applied to TQM research apply equally here.
“It is maddeningly difficult to do such research well, for several reasons:
First, there are serious measurement problems associated with even standard indices of firm performance such as market share, profitability or stock price (Brief, 1984; Penning, 1984; Kaplan and Norton, 1992)…
Second…Exogenous disturbances can significantly obscure the link between work processes and organizational outcomes…
Third, temporal issues can obscure intervention-outcome relationships (Whitten and Cameron, 1994)…
Taken together, these three difficulties can make it nearly impossible to detect statistically the direct effects of TQM on global measures of organizational outcomes”
(Hackman and Wageman, 1995, Section: What Outcomes are Attained, ¶ 2-5)
Another temptation is to identify the value of different independent activities or initiatives associated with implementing project management (such as training or project management maturity or the value of any one project) and extrapolate from there. Several recent studies addressed project success factors. These studies include success factors on projects, project management, and the organization (Cooke-Davies, 2002b), the influence of project management offices (PMOs) on reported project performance (Dai, 2002; Dai & Wells, 2004), the impact of project management software acceptance on project success (Bani-Ali & Anbari, 2004), and the association of project risk management practices with reported project success (Voetsch, Cioffi, & Anbari, 2004). Despite their importance, these studies do not result in the measure of the overall value of project management implementation as the approaches do not hold constant the impact of all other PM or organizational initiatives taking place in the organization at the same time.
In reviewing the ROI research in IT, TQM, Expert Systems, and Human Resources discussed above, it became clear that the conceptual model underlying this study must incorporate appropriate measures of what has been implemented in each organization, the process and outcome impacts of this implementation, the fit between what was implemented and the business orientation and environment within which the organization functions and appropriate measures of the benefits and costs of the PM implementation. The following diagram reflects initial thinking about the nature of the relationships that need to be examined in order to collect appropriate data to conduct ROI research.
Exhibit 1 – Preliminary Conceptual Model
First we need to distinguish between the rhetoric and the actuality of what has been implemented and the fit between these initiatives and the organizational and business environment. The choice of what will have been implemented (the PM Implementation in the context of any one organization) will be influenced by the business orientation of the organization – its focus, strategic direction and vision of itself as an entity – and the environment within which the organization operates – which will be influenced by its industry, customers, economic context and the types of projects the organization typically manages.
In understanding the impact of project management to this organization, there are three direct influences that will govern whether value is actually being realized:
- Firstly, the fit of what has actually been implemented needs to be understood in the context of the business orientation and the environment; in other words, to what degree did the organization ‘get it right' in establishing a context of project management that is appropriate for them and the types of projects they manage?
- Secondly, what are the Process Criteria of value – in other words, to what degree does this framework better influence the delivery of projects? Are the processes more efficient, more effective or more capable of delivering projects more reliably?
- Thirdly, what are the Outcome Criteria of value – in other words, to what degree do these project management capabilities actually deliver a bottom-line impact in terms of reduced costs, optimized efficiency or increased revenue? What is the return to the organization for investing in the project management capabilities it has established?
The following diagram identifies the relationships that can occur between the components of project management and the accrued benefit to the stakeholder(s). This diagram suggests how different attributes can provide a starting point for the elaboration and identification of the relationships and variables that need to be measured and statistically explored in this study.
Each case study will identify which capabilities and attributes exist within the organization from a normalized but evolutionary framework of capabilities that define the full set of practices and attributes seen in organizations. This framework will be developed with inputs from the initial literature search, the participants (subject matter experts, academics and executives), and the pilot case studies.
The capabilities of each case study will be associated with the values that are being demonstrated within that case study, and correlated with the results of each other case study conducted. In this way, it is expected to be able demonstrate the holistic delivery of value through discrete project management capabilities. Further, it is expected that this holistic value will be realized through specific and identifiable clusters of practice that are consistent and verifiable – and therefore generalizable – through the other cases and practitioner inputs.
As well as collecting cases that are geographically and industry sector diverse, it is important to distinguish between the different types of projects each organization carries out. Rather than do this at a fine grain level, we are proposing at this time using a three level taxonomy of: Business Projects (done by one company for another), Development projects (like innovation, NPD or R&D) and Change Projects (IT, Organizational Change and TQM). (Soderlund, 2004). This segmentation would be used to differentiate between project information collected between as well as within organizations. The projects within each case study would be classified by project type, and the relative value of the projects and project management for each of these project types – as well as the overall value of project management within that organization – can thus be established. While this framework is provisional, and will be revisited as the initial data collection requirements for the pilot case studies is finalized, it provides a framework that again contributes to ensuring full generalizability of the study findings.
Exhibit 2 – Relationship of Project Management Capabilities & Demonstrated Value
Potential Barriers to Success and Mitigation Strategies
A complex, integrated project of this nature must be considered a high risk undertaking and so is being designed as such. In this section we review the key risks and barriers to success of undertaking this research study and our mitigation strategies for each.
Ensuring credible, reliable, valid results
Initiatives to measure ROI often fail because they focus only on measuring the quantifiable costs and benefits and ignore the holistic evaluation of what was implemented, how well it was implemented and any intangible benefits that result. They also try to avoid estimating values and so stick with only the easily quantified cost and benefits. On the other extreme, ROI studies sometimes fail because of using overly optimistic estimates of benefits or underestimates of costs, resulting in low credibility of the results.
Since organizations do not necessarily account for expenditures or measure outcomes in the same way, any study seeking to be able to compare findings across organizations must be prepared to deal with this issue and normalize both the measures used and the results being reported. Our study will use the following tactics to address the development of credible and isolatable results.
Credible means believable, convincing, probable and trustworthy. Valid means that another researcher could replicate the study and achieve the same results. To ensure the credibility and validity of our results we will:
- Continually check both our process and our results with academic and practitioner advisory groups and the participant organizations.
- Devise data collection instruments and procedures that allow all the research teams to collect consistent data across organizations.
- Use only credible sources for collecting and analyzing data and use triangulation (more then 1 data source wherever possible).
- Where there is a choice among alternative valuations, always use the most conservative and document the choice.
- Ignore extreme data items and unsupported claims.
- Confirm all calculations with all stakeholders in each organization.
These measures combined with the efforts of a highly credible group of researchers consciously engaging in rigorous research activities from a variety of perspectives should result in highly credible and valid ROI measures.
Isolating the PM Impacts
The question of isolation refers to efforts to identify the impacts of the project management initiative separate from all the other initiatives taking place in the organization at the same time, as well as other environmental and situational factors. The risk is that the research will not be able to unequivocally state which benefits are attributable to the PM implementation.
We will use the following methods (as suggested by Philips, 1998 & 2002) wherever possible to isolate the project management benefits from other initiatives:
- Trend lines projecting the value of specific output variables if the PM initiative had not been implemented will be compared to actual data after the implementation.
- Organizational estimates of the total amount of improvement on key variables, and the apportionment or ‘adjustment’ necessary to reflect the portion of the improvement attributable to the PM initiative.
- All extraneous factors will be identified and their impact estimated. What remains will be attributable to the PM initiative. Dummy variables will be used to account for their level of input on cross organizations analysis.
- An understanding of the organizational environment and context (both internally and externally) will be monitored throughout the periods of study to understand and triangulate the impacts that internal initiatives as well as external market, economic and competitive factors have in influencing the observed results.
Sufficient data to make valid generalizable inferences
While we strongly believe that in depth analysis of a controlled subset of organizations is essential in order to arrive at holistic and justifiable results, we also recognize that this potentially opens up the research to criticism as to the applicability and generalizability of the findings beyond the scope of the industries and types of projects evaluated. Conducting 50 detailed case studies should provide for sufficiently generalizable results and allow for statistical analysis of relationships. In addition, we propose a more comprehensive web based data collection effort aimed at gathering data from 200 + organizations if deemed appropriate. This effort would allow us to verify and validate the local findings of the case studies and confirm their applicability in a more general context.
Agreement and validity of variable measurement
Measurement of the key variables will be an important challenge to this study. Selecting any one approach to measuring value risks criticisms from various disciplines as to both the definition and measurements of key variables, not to mention the validity of the analysis techniques. We have designed the research to include a wide array of perspectives in order to, in each instance, be able to triangulate results through more than one instrument or assessment technique. In this manner, not only should we be able to ensure that statistically valid correlations and qualitatively rigorous relationships are identified, but also that the correlations and relationships exist across multiple perspectives and inputs.
Bani-Ali, A. S. and Anbari, F. T. (2004) Project management software acceptance and its impact on project success Proceedings of the International Research Network on Organizing by Projects (IRNOP VI), Turku, Finland.
Barua, A, Lee, CHS, and Whinston TA. (1996) The calculus of re-engineering. Information Systems Research. 7 (4) p 409.
Barua, A, and Mukhopadhyay, T. (2000) Information technology and business performance: past, present and future? M: 2mud, R Wed. Framing the Domains of it Research: Projecting the future through the past. Cincinnatti, Ohio: Pinnaflex Educational Resources. P 65-84.
Cooke-Davies, Terence (2002a) Establishing the link between Project Management Practises and Success Proceedings of the PMI Research Conference 2002.
Cooke-Davies T. J. (2002b) The “real” success factors on projects International Journal of Project Management, Volume XX, pp 185-190.
Crosby, (1979). Quality is free. New York: McGraw Hill.
Dai C. X. (2002) The Role of the Project Management Office in Achieving Project Success, Doctoral Dissertation, The George Washington University, Washington, DC.
Dai C. X., & Wells, W. G. (2004) An exploration of project management office features and their relationship to project performance International Journal of Project Management, 22(7) 523-532.
Easterby, M. & Danusia, M. (1999) Cross Cultural collaborative research: Toward reflexivity. Academy of Management Journal. 42(1) p76.
Eriksson, H. & Hansson, J. (2003) The impact of TZM on financial performance Measuring Business Excellence. 7(1) 36-49.
Hackman, J.R. & Wageman, R. (1995) Total Quality Management: Empirical, conceptual and practical issues Administrative Sciences Quarterly. 40(2) 309-342.
Kwak, Y.H. & Ibbs, W. C. (2000) Calculating project management's return on investment Project Management Journal. 31(2) 38-47.
Ibbs, C.W., Reginato, J. & Kwak, Y.H. (2004). Developing project management capability – benchmarking, maturity, modeling, gap analyses, ROI studies. In Morris, P.W.G. & Pinto, J.K. (Eds) Chapter 48 The Wiley Guide to Managing Projects. John Wiley & Sons.
Montes, F.J.L., Jover, A.V.,& Fernandez, L.M.M. (2003)Factors affecting the relationship between total quality management and organizational performance The International Journal of Quality and Reliability Management. 20(2) p 188.
Mottiwalla, I & Fairfield–Sonn, J. (1998) Measuring the impact of Expert Systems. Journal of Business and Economic Studies 4(2), p1-17
Mullaly, M. (2004) Executive Summary of the Organizational Project Management Baseline Study 2003”, Edmonton, AB: Interthink Consulting Incorporated. Available at http://www.interthink.ca/research/home.html
Philips, JJ. (1998) Measuring the Return on Investment in Organization Development. Organization Development Journal. 16(4) p 29.
Philips, J. J. (2002) Measuring Return on Investment in Government: Issues and Providers. Public Personnel Management. 31(2) p 225.
Phillips, P.P. and Philips JIF (2004) Rot in the Public Sartor: Myths and Realities. Public Personnel Management. 33(2) 139
Powell, T.C. & Dent-Micallef, A (1997) Information technology as competitive advantage: The role of human, business, and technology resources. Strategic Management Journal. 18(5) 375-405.
Reginato, J. & Ibbs. W. (2002) “Project management as a core competency” Proceedings of the PMI Research Conference. Seattle, WA
Soderlund, J. On the evolution of project competence: Empirical regularities in four Swedish Frims. Proceedings of IRNOP vI, Torku, Finland, August 25-27, 2004.
Tuttle, T. & Romankowski, J. (1985) Assessing Performance and Productivity in white collar organizations. National Productivity Review 5 211-224.
Voetsch, R. J., Cioffi, D. F., & Anbari, F. T. (2004) Project risk management practices and their association with reported project success Proceedings of the International Research Network on Organizing by Projects (IRNOP VI), Turku, Finland.
Zbaracki, M. (1998) Rhetoric and Reality of Total Quality Management Administrative Sciences Quarterly 43(3) pp 602-636.
Copyright © 2005 Janice Thomas, Mark Mullaly
Originally published as a part of 2005 PMI Global Congress Proceedings – Toronto, Canada