Defining the value of project management
in search of value
For some time, investments in improving project management maturity have been promoted as an enabler of organizational efficiency, competitiveness and potentially a source of long-term strategic advantage. However, the benefits of these interventions have proven difficult to identify and quantify on a consistent basis. Many organizations now struggle to justify continued or new investment in project management initiatives. The financial benefit of improved project management practices, which were assumed as a matter of faith, are now being seriously questioned by senior executives who continue to be bombarded with highly publicized project failures.
In this increasingly results-oriented environment, project management champions are now required to document and justify the value received for this investment. Often the demand is for some form of financial justification—often in the form of a return-on-investment (ROI) calculation. Ultimately, key stakeholders want to understand what they are getting for this investment—how and to what extent these investments are contributing to organizational success.
This is not a challenge faced only by project management practitioners. Organizations are asking for, and even demanding, that a wide variety of organizational interventions ranging from teaming to performance appraisal and human resource management, TQM interventions, and IT infrastructures and expert systems (Barua & Mukhopadhyay, 2000; Eriksson & Hansson 2003; Montes, Jover & Fernandez 2003; Mottiwalla & Fairfield–Sonn, 1998; Philips, 1998; Philips, 2002) be evaluated and justified. However, this demand for evaluation presents a significant challenge in the project management world where our understanding of the cost and benefits of these interventions is at a very early stage of development. Research into how project management interventions translate into beneficial organizational outcomes is limited (see papers at this conference for some up-to-date results). Until recently the focus has been almost entirely on the conceptually simple approach of attempting to calculate the practically difficult concept of ROI.
This paper provides a theoretically grounded and conceptually holistic foundation for a major research effort initiated to provide answers to these important and some would say urgent questions. We begin by exploring the challenges of measuring organizational value through a review of the main approaches that have been used, with particular reference to their use in the project management arena over the last few years. Next, we discuss a theoretically grounded, holistic, and systemic model of value evaluation. We conclude with a discussion of the research design we are using to test this approach and some of the challenges facing the measurement of value in this way.
Limitations of Measuring Organizational Value
Efforts to determine the value of organizational activities has been a long-standing pursuit in a number of disciplines. The conceptual approaches to the value question can be roughly categorized as follows:
- ROI approaches
- Balanced scorecard (BSC) metrics
- Organizational competency approaches.
Each of these approaches is briefly described next, the project management efforts to evaluate value using there approaches are summarized, and the section concludes with a summary of the advantages and disadvantages of each of these approaches.
Measuring ROI stems from a time when it was easy to base the value of an enterprise on the value of their financial assets (things like revenue, real property, or equipment). The assumption is that for every dollar invested, there is a directly correlated financial return attributable to that investment. For instance, if a $1 investment in more manufacturing capacity results in 50¢ additional profit, resulting in 50% ROI, it is relatively easy to decide whether to invest in more capacity. This industrial view of valuation is not as useful in information- and service-based industries, or the so called knowledge economy, where it is commonplace for a company to be valued as much for its intangibles—its business processes, customer lists, trademarks and patents, knowledge, skills and business relationships—as for its financial and physical assets. Unfortunately, these value generators are often seen in combination and are much more difficult to individually measure and comprehend.
There are two problems this creates for ROI calculations. First, ROI calculations are more useful when it is possible to directly track the influence of the investment on the output, as in the manufacturing example. When the investment is in many activities, it becomes very difficult—some would say virtually impossible—to isolate and attribute a particular business result to a specific organizational initiative because so many of there variables can effect the result. As one executive put it, “Calculating an ROI of project management wouldn’t be very useful to me. Project Management is too many things. Even if I had an ROI, I still wouldn’t know what specifically to invest in.”
Nevertheless, many still feel that these financial measurements of value are fundamentally important. We have lumped three types of measures under the ROI label because they deal specifically with financial value. Each is described next along with references to project management literature either advocating or using this approach.
Benefit/Cost Ratio (B/C)
The benefit/cost ratio is a simple calculation of the ratio of the benefits directly attributable to project management divided by the costs directly associated with project management.
B/C Ratio = $Benefits/ $Costs
The advantage of this method is that it provides a fairly clear dollar value of benefit earned for every dollar invested. Disadvantages of this method are that nonfinancial-valued benefits are given short thrift or must be approximated in some way and this can lead to undervaluing the benefits of certain types of investments. In addition, the ratio method can result in higher ratio returns for investments that have a lower overall dollar amount benefit. Both Knutsen (1999) and Smith and Barker (1999) discussed the use of this approach in a project management context.
Return on Investment (ROI)
ROI is a similar calculation. Here the net benefits are divided by the total costs and the result is multiplied by 100 to give a percentage return for each dollar invested.
ROI = ($Benefits-$Costs)/ $Costs X 100
Proponents of evaluating and measuring the ROI of organizational initiatives state that measuring ROI is neither a new, nor particularly complex managerial tool. Ultimately, it is simply a cost benefit analysis method for comparing the costs of an initiative with its benefits. Proponents (Phillips & Philips, 2004) stated that the biggest challenges of conducting ROI evaluations relate to ensuring credible, valid results in a reasonable time frame; demonstrating that the results exhibited are appropriately apportioned and attributable to the program or initiative being investigated; and isolating the impacts of the program or initiative from other context-specific and situational factors. The underlying message is that this is not a terribly difficult thing to do.
A disadvantage of this approach is that it fails to take into account the cost of capital and the time value of money. Sometimes these methods are adjusted as follows to take the time value of money into account.
B/C Ratio = NPV Revenues/ NPV Costs
However, the difficulty in developing dollar values for both the costs and benefits that are credible, and ensuring that intangible benefits are considered, remain.
Maturity Based ROI Metrics
The work of the team of Ibbs, Kwak, and Reginato (Ibbs, 2000; Ibbs & Kwak, 1997; Ibbs & Kwak, 2000; Ibbs & Reginato, 2002; Ibbs, Reginato & Kwak., 2004; Kwak & Ibbs, 2000; Kwak & Ibbs, 2002; Reginato, 2002) over the last 10 years focused on recognizing the benefits of investment in project management competency through measures of maturity in an organization’s practice of project management. In this way their work uses both ROI and benchmarking to value the investment in project management. Higher maturity scores are hypothesized to correlate with higher levels of predicted project performance. The theory was that investment in project management increases a firm’s project management maturity standing and this investment results in improved project performance that should translate into cost saving and other benefits. They calculated the ROI of increasing an organization’s project management maturity as the change in annual project revenues divided by annualized project costs as follows.
PM/ROI = (Predicted Profit Margin—Current Profit Margin) X Annual Project Revenues
Annualized PM Expenditures
Results of the two major empirical tests of this theory were inconclusive. Although an association between higher levels of project management maturity and project cost performance and project schedule performance were recognized they did not have enough data to deliver statistically significant findings. In addition, their findings were extrapolated from the performance of one project. Another flaw in this measure is that it associates no value with finishing the project faster or with customer satisfaction or quality improvement benefits.
Others have contributed to this stream of research (Crawford & Pennypacker, 2001; Pennypacker & Grant, 2003), through survey-based research attempting to associate firm self report data on maturity measures and project performance. Although flawed, these approaches provide some interesting insights that deserve further exploration.
Criticisms of ROI Approaches in General
All ROI approaches are open to the criticism that these forms of metrics ignore or discount the human or intangible benefits that come from improving organizational functions. As Githens (1998) pointed out in the project management context these metrics are criticized for their role in preserving the status quo, emphasizing capital investment and returns over intellectual and yielding predictable returns. Cabanis-Brewin (2000) echoed these criticisms in stating that these methods do not recognize the value of the cultural change and “soft” human factor benefits. Jugdev and Thomas (2002) made a clear case that easily copied competencies do not yield the long-term strategic advantages that today’s organizations are looking for from their investments.
There is no question that bottom-line results are important for all types of organizations regardless of their profit orientation. All organizations need to pay attention to revenue and costs to remain financially viable. However, not everything that an organization does can be translated into monetary terms. For instance, the link between employee satisfaction and corporate performance is difficult if not impossible to establish. However, many organizations believe that keeping employees satisfied is an important corporate goal. In not-for-profit or government organizations, measuring objectives in financial terms is even more problematic. Translating employee or customer satisfaction into a dollar value or evaluating outcomes in financial terms is very difficult to calculate in any valid or credible way. Thus, despite the perceived desire for an ROI calculation for such organizational interventions, most executives are often leery to accept those calculations that are presented.
In addition, some of the companies that have invested heavily in project management over the last decade are now questioning and/or decreasing their levels of investment. It seems that their experience, as our common sense, would suggest that there are diminishing returns to investments in project management. Improving project management practices improves project and organizational performance to a point, but past that point justifying continued investment becomes difficult. In traditional project management industries like construction and consulting these improvements may translate directly into cost reductions or revenue increases that compensate for the ongoing investments. However, as project management becomes a more common method of working in a larger array of organizations, and projects, these standardized practices may be less useful, particularly in those industries that are more focused on knowledge work.
Given the difficulties and constraints of using ROI to measure value in an organizational setting, many researchers have sought to develop more sophisticated measures. By far the most used of these expanded approaches is the balanced scorecard (BSC) approach developed by Kaplan and Norton (2000a; 2000b). Both Githens (1998) and CabanisBerwin (2000) advocated a BSC approach as a response to the challenges previously discussed. We turn to examine how it has been used in the project management field and its recognized weaknesses next.
Balanced Scorecard Metrics
Balanced scorecard (BSC) metrics try to evaluate organizational performance using a variety of financial and nonfinancial measures including the following:
- Learning and growth
- Internal measures
- Customer perspectives
- Financial perspectives (such as the ROI).
In particular, this approach attempts to measure the knowledge based and intangible benefits associated with organizational effectiveness today. Webber and Simsarian (2004) gave a thorough review of the application of this approach. BSC metrics have the advantage of attempting to evaluate all the benefits and costs of each organizational action in the context of the specific organization’s strategy. The selection of appropriate metrics is therefore key to the success of this approach.
In Europe, a similar model has been developed based on this approach in combination with quality perspectives. This model, developed by the European Foundation for Quality Management (EFQM), seeks also to evaluate organizational performance based on a wider variety of potential measures.
Building from the EFQM model, Westerveld (2003) defined a project excellence model and Bryde (2003) derived a project management performance assessment model to be used to evaluate the performance and contribution of project management within the organization. Bryde published the results of his survey as an effort to explore the value of project management in a way that built on the structure of the EFQM model. The study intent and constructs are interesting but the small sample size and weakness of construct validity limits the usefulness of the study.
The criticisms of these approaches are also many. Ittner and Larcker (2003) pointed out that many people in attempting to apply the BSC approach fail to tie the metrics back to corporate strategy—often because commonly available or easy-to-acquire measures substitute for the more difficult but appropriate strategic measures. For example, although Crawford and Pennypacker (2001) advocated the use of a “balanced family” of metrics associated with shareholders, employees, and communities, they did not discuss how to select and quantify these metrics or link them to corporate strategy. They appeared to be falling into the trap that Itner and Larcker report of failing to link measures to strategy, setting faulty performance targets, and measuring the targets ineffectively. In addition, attempts to quantify intangible benefits fall prey to criticisms of how they are estimated. Crawford and Pennypacker’s empirical work using this approach also falls prey to the common issue that organizations do not often collect or maintain the kinds of information necessary to do this kind of evaluation.
In addition, while many authors show how the BSC approach focuses an organization on improving measurable performance in order to optimize operational efficiency (Bontis, Dragonetti, Jacobson & Roos, 1999; Roos, Roos,Dragonetti, Edvinsson, 1997; Russ, 2001), Voepel, Leibold, Eckhoff, and Davenport (2006) argued that this single-minded focus on a small number of relatively rigid measures results in a tyranny of measurement that conflicts with creativity, innovation, and adaptation. Likewise, they argued that the internal focus of the BSC approach encourages organizations to ignore external circumstances, often to their own detriment.
Finally, executive management often sees estimates of these metrics as speculative and debatable rather than being the hard numbers they are looking for. In addition, some of the financial measures that are recommended here—like earnings per share (Crawford & Pennypacker, 2001) are notoriously difficult to associate with individual organizational initiatives as there are typically too many other activities and initiatives in organizations at the same time to be able to tie share prices to improvements in one specific area versus another.
Organizational Competency Approaches
Emerging at roughly the same time as the BSC approach, the competency based perspective of strategic thought emphasizes the impact that internal organizational competencies have in determining the long-term, sustainable competitive advantage of firms. According to this perspective, each firm develops a unique combination of corporate assets and capabilities that allow it to generate income based on the exploitation of these competencies (Barney, 1991; Grant, 1991; Peteraf, 1993; Wernerfelt, 1984). Strategic competencies are those that contribute to sustainable competitive advantage for firms. Competencies can be both the abilities and specific skills that a firm possesses or the cognitive characteristics that allow them to deploy them in a specific way. The common understanding is that these firm specific assets and competencies are knowledge related, tacit, difficult to trade, and typically shared among the agents of the firm. Thus, the question emerges as to whether or not project management could be a capability capable of generating long-term competitive advantage to a firm.
Jugdev (2002; Jugdev & Thomas, 2002) has made the most rigorous attempt to evaluate project management’s capacity to generate long-term competitive advantage. This research looks at the formation and enhancement of project management capabilities through the use of maturity models as a framework for competency development. Her conceptual arguments assert that maturity models do not in themselves generate advantage as they are easily copyable. Her empirical findings from a small sample exploratory study suggest that project management may be an enabler rather than a strategic asset. However, this research remains to be tested with larger-scale samples. One of the problems with all competency-based empirical research has been in operationalizing the attributes of interest and synthesizing appropriate measures.
Literature in the TQM field may provide some insights into how to do this. Similar to project management, literature suggests that quality management initiatives can contribute to sustainable competitive advantage “by encouraging the development of competencies that are specific, produce socially complex relationships, are imbued in the history and culture of the organization, and generate tacit knowledge” (Escrig-Tena & Bou-Llusar, 2005). Building from previous literature classifying four main types of competencies: managerial, input-based, transformation-based, and output-based (Lado, Boyd & Wright., 1992; Lado & Wilson, 1994), Escrig-Tena and Bou-Llusar (2005) developed a method of operationalizing these types of competencies in the context of a TQM initiative that may prove very useful for evaluating the contributions of project management initiatives.
Valuing organizational initiatives is a difficult activity fraught with challenges (see Thomas and Mullaly  for a review). Voelpel et al. (2006) further asserted that “All of the traditional business performance measures suffer to some degree because of the underlying and increasingly invalid assumptions rooted in the industrial economy.” They suggested that what is missing in these measures is a contextual understanding of the complex web of interrelated factors, relationships, and activities that need to be taken into account in a holistic manner in order to assess an organization’s performance in the knowledge economy.
Given that current approaches to measuring the value of organizational initiatives face such limitations, how then do we identify and calculate the value an initiative like project management delivers to an organization? What is needed is a way for managers to decide between alternative investments in organizational initiatives, enabling them to choose the initiative that is likely to yield the greatest return for limited resources within their particular context. We need to be able to identity what types of project management investments are most likely to result in the needed benefits and outcomes for a specific organization at a specific point in time. This requires deep, rich data on how an organization functions, what it has invested in, and the results it has received collected in such away that the organizations can be compared both on what is similar across them and what is dissimilar.
The next section describes an initial conceptual model designed to help us understand how project management creates value in today’s organizations to serve as a basis for data collection.
Organizational innovation literature studies the ways that organizations adopt initiatives that change the way they do business. These change initiatives can be internally generated or arise as the result of external pressures or trends (Damanpour, 1996; Damanpour & Evan 1984). Often organizational changes arise as trends and ideas are “borrowed” from one organization or industry to support innovation in another (Abrahamson, 1991). Organizational innovation has taken many forms ranging from new products and services or process technologies to organizational structures, administrative or computerized systems and other ways of working. Examples of organizational change initiatives that have been studied extensively include: the introduction of matrix organizational structures (Burns & Wholey, 1993; Mahajan, Sharma et al., 1988; Teece, 1980;); TQM systems (Guler, Guillen & Macpherson. 2002; Westphal, Gulati & Shortell., 1996; Westphal, Gulati & Shortell, 1997); and data processing and information technology (Kimberly & Evanisko, 1981). Project management initiatives clearly deserve similar careful study, as they tend also to influence both the technical and social systems of an organization through changes to structures, work practices, technical systems, communication, and behavioral patterns.
Institutional theory studies the generation, development, and implementation of ideas across organizations (see for example DiMaggio and Powell (1983). Research in this area suggests that organizations that innovate early are more likely to customize the innovation to their particular circumstances and thereby receive greater organizational efficiency benefits than late adopters (Westphal et al., 1997). Late adopters are often strongly encouraged to adopt the processes developed by early adopters and so are more likely to take on practices that do not fit their organization and less likely to reap the benefits of early adopters (DiMaggio & Powell). This literature suggests that those that adopt earlier or who customize the innovation to fit within their environmental and organizational context are more likely lo receive higher rewards from the innovation. In addition, this literature recognizes at least two primary benefits from implementing new organizational practices. Some organizations seek to maintain or increase their performance through improvements in efficiency or effectiveness (Damanpour, 1987), whereas others seek to adopt such practices to increase their legitimacy and social fitness regardless of the impact of the adopted practices (Meyer & Rowan, 1991; Westphal, 1997). Thus, understanding why and when project management was implemented is likely important to understanding how it is implemented and what the results are likely to be.
A further subset of this literature seeks to determine the “value” these results deliver to the organizations that adopt them. Organizational initiatives of all types share three particular features in common. First, the end result or “product” of the initiative is not as clear as in many other innovations. Second, the incentives, benefits, and results of adopting them are often difficult to discern (Alange, Jacobsson & Jarnehammer., 1998) and quantify. Third, research suggests that it is difficult if not impossible to imitate these innovations in any way likely to produce the some benefits due to the organization-specific nature of these implementation efforts and the local interpretation and innovations necessary to attain success (Mahajan et al., 1988; Teece, 1980). In addition, regardless of the ability of either scholars or practitioners to quantify initiatives’ value, there are no lack of articles proclaiming their strategic importance to the success and even survival of organization. Work like Powell’s (1995) examination of TQM, which proposes its contribution to sustained competitive advantage, lays the foundation for the work of Jugdev (2004) and others exploring project management’s contribution to longer-term competitive advantage.
Organizational change literature examines the process of change involved in implementing organizational innovations. Although there tends to be a pro-innovation bias in most western cultures that assumes that innovation is always successfully implemental and result in beneficent charges (Abrahamson, 1991), this literature clearly shows that while a worthwhile change initiative may be identified and launched, the expected benefits may never materialize. Sometimes other important and valuable benefits are realized—sometimes the initiative is a bust (Hinings & Greenwood, 1988). Determining whether the innovation has actually been adopted can be difficult in and of itself (Damanpour 1987; Damanpour & Evan 1984; Zbaracki, 1998). Sometimes innovations are rejected and sometimes innovations do not actually result in improvement (Abrahamson, 1991). Thus, in trying to determine the value of an innovation it is important to explore what has actually changed and why. Understanding what the project management initiative has accomplished requires more than asking individuals. It requires both asking individuals and exploring these “espoused” theories through a process of observation and interpretation to deduce what the “theories-in-use” actually are (Argyris & Schön, 1978).
Clearly any study designed to determine the value organizations receive from implementing project management based initiatives must be deeply rooted in this rich stream of organizational research and in the experience of individual organizations. In understanding the impact of project management on an individual organization, there are three direct influences that will govern whether value is actually being realized:
- First, the fit of what has actually been implemented needs to be understood in the context of the business orientation and the environment; in other words, to what degree did the organization “get it right” in establishing a context of project management that is appropriate for them and the types of projects they manage? Generalizable value statements can only come from this level of rich understanding of individual organizations.
- Second, what are the process criteria of value—in other words, to what degree does this framework better influence the delivery of projects? Are the processes more efficient, more effective, or more capable of delivering projects more reliably?
- Third, what are the outcome criteria of value—in other words, to what degree do these project management capabilities actually deliver a bottom-line impact in terms of reduced costs, optimized efficiency, or increased revenue? What is the return to the organization for investing in the project management capabilities it has established?
Figure 1 hypothesizes the relationships that can occur between the components of project management and the accrued benefit to the stakeholder(s). This diagram suggests how different attributes can provide a starting point for the elaboration and identification of the relationships and variables that need to be measured and statistically explored in this study.
Each of these constructs is explored in some detail in the following sections.
Figure 1. Understanding the Value of Project Management
Note: Developed from (Thomas & Mullaly, 2005)
Bruner (1990) asserted that it is impossible to understand the metrics and reference system of a company without first understanding the situated, contextual interpretations embedded within the management practices of the company. The conceptual model we defined first recognizes that there is an external context that influences the managerial practices within each organization and ultimately the success of any organizational initiative. This is the context within which the initiative is being launched and takes into account the organizational, strategic, and economic contexts. Although not always readily apparent, these variables are likely to influence the results of the project management initiative in at least two ways. First, if the project management initiative does not “fit” with the organization, strategic, or competitive environment, it is unlikely to deliver desired results, Second, something else going on in the organization may weaken, jeopardize, or overstate the potential benefits from the project management initiative. Without understanding the context, it is impossible to know what other organizational or environmental activities may be influencing your results.
In order to understand the organizational context, we need to explore each of the following in some detail:
- Strategic context—customers, suppliers, strategy
- Economic/political context—location, politics, strength of economy
- Organizational attributes—people, projects, culture, and infrastructure.
The choice of what will have been implemented (the project management implementation in the context of any one organization) will be influenced by the business orientation of the organization—its focus, strategic direction, and vision of itself as an entity—and the environment within which the organization operates. This in turn will be influenced by its industry, customers, economic context, and the types of projects the organization typically manages. Clearly careful study of these contextual variables goes a long way to addressing Voepel et al.’s criticism of metrics like the BSC that focus solely on the internal performance impacts of organizational initiatives.
Project Management Constructs
There are many project management or organizational project management maturity models available to choose from on the market. Many readers may be asking at this point why one of them was not chosen for use in evaluating this aspect of the model—especially as at least two of the project team members own proprietary maturity models and PMI® is sponsoring the study, so why not use the OPM3® model?
Truthfully, each of these models (where we had access to them) was evaluated as potential instruments for use in this study. What we found was that none of them had the depth and breadth of variables we felt were important to include in order to evaluate all aspects of the project management initiative. Each of these instruments was created for a specific purpose and to shed light on some aspects of project management or performance but in doing so also often left others in the dark. As per Callon’s (1990) insights, each set of predominant metrics serve to identify certain activities and initiatives as essential or detrimental to the advancement and development of an organization or function. In particular, most of the existing instruments focused on the tangible elements of project management, ignoring or underconsidering the intangible and innovative capabilities that are necessary to manage in a high-uncertainty, high-ambiguity knowledge-based economy.
In addition, each of the instruments underlying these models on their own ran to a large number of questions or items. Given the scope of the data we needed to collect from each organization we had to trade off the use of an established instrument alongside our other data collection requirements against the willingness of organizational participants to complete the instrument.
In the end we chose to develop instruments that were as much as possible based on published research and items that had been tested in previous studies. We also chose to focus specifically on the data collection of what we needed for this study rather than undertaking the effort to verify or endorse any particular maturity model. Rather than pursue specific evidence of project management knowledge and practice as defined within any specific instrument, we chose to develop a more detailed understanding of the processes through which practice and knowledge claims are identified and created in the process of becoming self-evident “best practices” in light of Latour’s (1987; 1999) work rather than look for any particular set of identified “best practices.”
We also recognized that we needed to do two things in collecting information on the project management implementation. First, we needed to collect information that would allow us to identify what has been implemented in the name of project management. To do this we needed to understand the following:
- The history of the project management implementation—who did what, when, why, and how?
- What organizational infrastructure has been created to support project management in the organization: What level of infrastructure support is there? Is there a PMO? What level does it report to? How many project managers are there? What is the budget for project management? What are the human resources policies? Etc.
- What project management practices have they implemented with respect to organizational integration, portfolio management, program management, project management, value realization, and resource management?
- What tools are used: software, guidelines, databases, reference sources?
- What are the project management human resources like?
- What training and development is offered to the project management personnel and what is the quality and intensity of this training?
Second, we needed to distinguish between the rhetoric and the actuality of what has been implemented (Zbaracki, 1998). That is, we needed to discern whether the policies that have been created are implemented or just sit in binders and are not used. To do this we needed to ask similar questions of different levels in the organization. We needed to ask people to tell us what the policy is and what they actually do to manage projects. We needed to compare this to the formal documentation of the policies and procedures that we can see in documents.
Finally, we needed some way to be able to assess the fit between these initiatives and the organizational and business environment. This information is likely to be reflected in analytical results derived by reviewing the statistical relationships between these variables. However, asking for a perceptual response to this question may also provide valuable insights.
In most organizations, expenditures on project management do not have direct impacts on revenue or profits. Although project management is often “sold” on an efficiency agenda, the fact is that project management improvements do not always reduce costs and can often increase them in the short run. Most project management improvements do not yield tangible revenue and cost impacts but more usually are associated with improving less tangible aspects of the project often related to meeting stakeholder expectations around cost, timing, quality, and process. This results in a number of different kinds of benefits for organizations.
Borrowing a long-standing evaluation framework from human resources development evaluation literature we propose to study five types of organization value.
- Level 1—Satisfaction
This is the simplest measure of value. Do the key stakeholders perceive that the project management initiatives provided value? This is measured through perceptions/self-report satisfaction levels as well as through the use of objective measures (such as repeat customers) wherever possible.
- Level 2—Aligned use of practices
This measures the fit between espoused theories and theories in use. Did the project management implementation result in the desired processes? Do you do what you say you do? Do project people know what they are supposed to do? This is assessed through a comparison of practices, policies, and procedures with what actually happens on projects.
- Level 3—Process outcomes
What project process improvements have you reaped from your project management implementation? How effective is the project management process? This is evidenced by changes over time in things such as numbers of change requests, budget performance, and reliability of delivery.
- Level 4—Business outcomes
What business outcomes are related to these process improvements? Improving project management can result in a number of different business outcomes depending on the nature of the organization. For organizations that do projects for clients for instance, improving project management practices may improve customer satisfaction and the organization’s ability to attract new customers through reputation effects, word of month, and potential advertising opportunities.
Research suggests that repeat business arises out of customer satisfaction and that even small increases in customer retention can have a dramatic impact on profits. For organizations that do manufacturing or research projects for product development, improving project delivery speed and reliability can improve an organization’s time to market performance, which has been shown to significantly improve organization and product performance. Organizations that do projects primarily for internal purposes such as organizational change and information technology projects can benefit from increased ability to achieve strategic goals.
- Level 5—Return on investment
For every dollar invested in the project management initiative, what return in terms of cost savings, revenue, etc., can be attributed to it? This measure is calculated based on the quantification of the direct business impacts identified in Level 4 benefits analysis as compared with the reported expenditures on project management. The reader might at this point ask why we bother with this calculation if ROI has the limitations noted earlier. One of the primary limitations of ROI is that many benefits have to be estimated and it is hard to see where the numbers are coming from. This study takes the approach that all the measures are rigorously documented and so the ROI in this case is at least based on solid organizational understanding and measures that can be used for the insights it provides, while still recognizing its inherent limitations.
Once each organization is analyzed based on the described approach, information from each organization will be coded for statistical analysis. Using factor analysis and other tools, we will look at this data in the hope of deriving relationships between categories of investment in project management and categories of benefits. Where we can identify these patterns, we are likely to identify specific benefits associated with developing specific project management capabilities. This will provide an analysis that is far more than a simple assessment of the average ROI that results from investing in project management. We should be able to show that investing in a particular form of project management provides a specific type of benefit in a specific context. This type of result will not simply provide value statements for project management. It will also provide guidance for those making the decision to invest in project management as to where best to spend their money to get the type of result they most desire and require.
Exploring this question in detail requires careful planning and assessment at multiple levels. Building from an extensive review of the literature on multiple case research, survey and organizational assessment design and construction, and on large-scale coordinated research projects similar to this one the project is designed to be carried out in two main phases. Each phase and its key activities are described next.
Phase 1—In Search of Value
The first phase entails drawing on a wide array of business and project management experts to develop the conceptual model previously elaborated and then to test this model and instruments through the use of five pilot studies. Each case study will identify which capabilities and attributes exist within the organization from a normalized but evolutionary framework of capabilities that define the full set of practices and attributes seen in organizations. This framework was developed with inputs from the initial literature search, the research teams (subject-matter experts, academics, and executives), and the pilot case studies. Data for each case study will be collected through personal interviews, document reviews, project file reviews, organizational surveys, library research, and financial analysis. Multiple response methodology is used to provide triangulation across methods and study participants and to include numerous perspectives on how the organization is perceived or rated on each measure. This approach is recommended by a wide number of articles have noted the need for using multiple instead of single respondents (Dawes, 2000; Gray, Mattear, Boshof, Maatheson., 1998; Tsai, 2002) to ensure that an organizational rather than individual picture is developed.
The capabilities of each case study will be associated with the values that are being demonstrated within that case study, and correlated with the results of each other case study conducted. In this way, it is expected to be able demonstrate the holistic delivery of value through discrete project management capabilities. Further, it is expected that this holistic value will be realized through specific and identifiable clusters of practice that are consistent and verifiable—and therefore generalizable—through the other cases and practitioner inputs.
As well as collecting cases that are geographically and industry-sector diverse, it is important to distinguish between the different types of projects each organization carries out. Rather than do this at a fine grain level, we are proposing at this time using a three-level taxonomy of business projects (done by one company for another), development projects (like innovation, NPD, or R&D), and change projects (IT, organizational change and TQM) (Soderlund, 2004). This segmentation would be used to differentiate between project information collected between as well as within organizations. The projects within each case study would be classified by project type, and the relative value of the projects and project management for each of these project types—as well as the overall value of project management within that organization—can thus be established. Although this framework is provisional, and will be revisited as the initial data collection requirements for the pilot case studies is finalized, it provides a framework that again contributes to ensuring full generalizability of the study findings.
This project requires the design and assessment of a large number of data collection and analysis instruments. Each organization will be providing a significant amount of their employee’s time and access to information to us. We won’t get a second chance. The pilot studies, and in fact all of phase 1, is designed to ensure that we have the right design, right instruments, and that the data we feel we need is available to us in most organizations.
Phase 2—Calculating Value
The second phase of the project entails developing approximately 15 research teams around the world who will collect data on four to five organizations each. Each team will be familiarized with the standard instruments that are derived from the pilot studies. In addition, each team will add a new dimension to the study by collecting data on their theoretically inspired research questions.
As each study is completed, specific data will be coded and entered into the statistical database for analysis by the stats team. Findings from the stats team will generate further analysis of cases and will inform the development of a survey instrument to use to collect self-report data from a larger number of organizations if it appears that this can be done in an appropriate manner.
Measuring the value of any organizational initiative is not a clear-cut endeavor. Many have tried it in the past, both in the project management arena and in others. Grand and Von Krogh (2006) recognized that “no calculation is possible without a metric and reference system, which identifies relevant entities, states of the world, possible actions and expected outcomes…In parallel we learn from the knowledge-based view of management that total framing is impossible: every representation of a company if we understand it as a knowledge system is fragile, problematic and context-dependent, but is nevertheless an important artifact for understanding managerial practice.” Thus, “studying and understanding the necessity and the impossibility of self-evident, unquestioned metric and reference systems for calculation in managerial practice, given uncertainty and ambiguity…” is a challenging exercise for those of us interested in improving practices while recognizing that the mechanistic perspective of earlier management research is not appropriate to the knowledge economy that we now operate in, and arguably never has been. Clear and uncontested metrics are problematic in a world that recognizes that the socially constructed, interpretivist, critical, and complexity-based approaches to management also provide strong insights into how to improve practices.
Clearly the failure of many of the earlier attempts to evaluate the value of new organizational initiative arises from two key problems: lack of access to data and a failure to examine data that is important to being able to address this question. In failing to recognize these contextual and interconnected operations as complex, socially adaptive systems, these earlier studies were not able to improve our understanding of these initiatives beyond a rudimentary level. This study has been designed to explore the issues and challenges of this type of research from as many research perspectives as possible to develop a much more sophisticated understanding of the value of project management to organizations.
We have reviewed literature across many fields, incorporated modern, post-modern, interpretivist, critical and complexity theorist (and many other) insights on measuring value, and developed a comprehensive and holistic model designed to address the failings of earlier studies and build from current best knowledge in many management fields. We believe we have a model and research design that will be able to answer questions that others have failed to either answer or address.
It is clear that these analyses are complex and will require the use of advanced statistical and qualitative analysis techniques. To be successful, we also require partner organizations that are willing to engage in extensive data collection efforts and provide access to data that in many cases is considered both confidential and strategic. Case studies will provide rich understanding of the process and value of project management in individual organizations. Cross-case analysis will provide insight into commonalities and differences, and develop testable hypotheses and research questions for further exploration. The validation studies will require structural equation modeling analysis to study the causal relationships hypothesized among the constructs. In short, the validation of this model will clearly be ambitious and demanding research. Nevertheless, if the network of causal influences on the outcomes of effective project management interventions are to be understood, this type of research is not just necessary and valuable, but essential.
By the time you read this we should have finished the pilot case studies and should be able to provide some insights into preliminary findings from these studies and any revisions or changes to the model based on the pilot experience.
Abrahamson, E. (1991). Managerial fads and fashions: The diffusion and rejection of innovations. Academy of Management Review, 16(3), 596–612.
Alange, S., Jacobsson, S., & Jarnehammer, A. (1998). Some aspects of an analytical framework for studying organizational innovations. Technology Analysis and Strategic Management, 10(1), 3–21.
Argyris, C., and Schön, D. (1978). Organizational learning: A theory of action approach. Reading, MA: Addison-Wesley.
Barney, J. B. (1991). Firm resources and sustained competitive advantage. Journal of Management, 17(1), 99–120.
Barua, A., & Mukhopadhyay, T. (2000). Information technology and business performance: Past, present and future? Framing the Domains of it Research: Projecting the future through the past (pp. 65–84). Cincinnati, OH: Pinnaflex Educational Resources.
Bontis, N., Dragonetti, N. C., Jacobsen, K. Roos, J. (1999). The knowledge toolbox: A review of the tools available to measure and manage intangible resources. European Management Journal, 17(4), 391–403.
Bruner, J. (1990). Acts of Meaning. Cambridge, Massachusetts, Harvard University Press.
Bryde, D. J. (2003). Modeling project management performance. The International Journal of Quality & Reliability Management, 20(2/3), 228–244.
Burns, L. R., & Wholey, D. R. (1993). Adoption and abandonment of matrix management programs: Effects of organizational characteristics and interorganizational networks. Academy of Management Journal, 36(1), 106–138.
Cabanis-Brewin, J. (2000). The elusive ROI. PM Network, April, 45–50.
Callon, M. (1990). Actor network theory—The market test. Actor Network theory and After (pp. 181–195). Oxford, England: Blackwell.
Crawford, K., & Pennypacker, J. (2001). The value of project management: Proof at last. PMI 32nd Annual Seminars & Symposium, Nashville, TN.
Damanpour, F. (1987). The adoption of technological, administrative, and ancillary innovations: Impact of organizational factors. Journal of Management, 13(4), 675–688.
Damanpour, F. (1996). Organizational complexity and innovation: Developing and testing multiple contingency models. Management Science, 42(5), 693–716.
Damanpour, F., & Evan, W. M. (1984). Organizational innovation and performance: The problem of “organizational lag.” Administrative Science Quarterly, 29, 392–409.
Dawes, J. (2000). “Market Orientation and Company Profitability: further evidence incorporating longitudinal data.” Australian Journal of Management 25(2): 173-199.
DiMaggio, P. J., & Powell, W. W. (1983). The iron cage revisited: Institutional isomophism and collective rationality in organizational fields. American Sociological Review, 48(2), 147–160.
Eriksson, H., & Hansson, J., 2003). The impact of TQM on financial performance. Measuring Business Excellence, 7(1), 36–49.
Escrig-Tena, A. B., & Bou-Llusar, J. C. (2005). A model for evaluating organizational competencies: An application in the context of a quality management initiative. Decision Sciences, 36(2), 221.
Githens, G. (1998). Financial models, right questions, good decisions. PM Network, 29–32.
Grand, S., & Von Krogh, G. (2006). Commentary on Voelpel et al.’s “The tyranny of the balanced scorecard in the innovation economy.” Journal of Intellectual Capital, 7(1), 59–60.
Grant, R. M. (1991). The resource-base theory of competitive advantage: Implication for strategy formulation. California Management Review (Spring), 15–21.
Gray, B., S. Matear, Boshof, C. & Maatheson, P. (1998). “Developing a Better Measure of Market Orientation.” European Journal of Marketing 32(9,10): 884-903.
Guler, L., Guillen, M., & Macpherson, (2002). Global competition, institutions, and the diffusion of organizational practices: The international spread of ISO 9000 quality certificates. Administrative Science Quarterly, 47, 207–232.
Hinings, C. R., & Greenwood, R. (1988). The dynamics of strategic change. Oxford: Basil Blackwell.
Ibbs, C. W. (2000). Measuring project management’s value: New directions for quantifying PM/ROI. PMI Research Conference, Paris, France.
Ibbs, C. W., & Kwak, Y. H. (1997). The benefits of project management: Financial and organizational rewards to corporations. Sylva, NC: Project Management Institute.
Ibbs, C. W., & Kwak, Y. H. (2000). Assessing project management maturity. Project Management Journal, 31(1), 32–43.
Ibbs, C. W., Reginato, J., & Kwak, Y.H. (2004). Developing project management capability—Benchmarking, maturity, modeling, gap analyses, ROI studies. The Wiley Guide to Managing Projects..
Ibbs, C. W., & Reginato, J. M. (2002). Quantifying the value of project management. Newtown Square, PA: Project Management Institute.
Ittner, C. D., & Larcker, D. F. (2003). Coming up short on nonfinancial performance measurement. Harvard Business Review, 81(11), 88–95.
Jugdev, K. (2002). Blueprint for value creation: Developing and sustaining a project management competitive advantage through the resource-based view. PMI Research Conference.
Jugdev, K., & Thomas, J. (2002). Project management maturity models: The silver bullet of competitive advantage? Project Management Journal, 33(4), 4–14.
Judgev, K. (2004) Project Management as a Strategic Asset. Unpublished Doctoral Thesis. University of Calgary.
Kaplan, R. S. and D. P. Norton (2000a). “Having Trouble with Your Strategy? Then Map It,.” Harvard Business Review 78(10).
Kaplan, R. S. and D. P. Norton (2000b). The Strategy-Focused Organization: How Balanced Scorecard Companies Thrive in the New Business Environment. Boston, MA, Harvard Business School Press.
Kimberly, J. R., & Evanisko, M. J. (1981). Organizational innovation: The influence of individual, organizational and contextual factors on hospital adoption of technological and administrative innovations. Academy of Management Journal, 24(4), 689–713.
Knutson, J. (1999). From making sense to making sense: Measuring project management ROI. PM Network.
Kwak, Y. H., & Ibbs, C. W. (2000). Calculating project management’s return on investment. Project Management Journal, 31(2), 38–47.
Kwak, Y. H., & Ibbs, C. W. (2002). Project management process maturity model. ASCE Journal of Management in Engineering, 18(3), 150–155.
Lado, A. Boyd, A., N. G., Wright, P. (1992). A competency-based model of sustained competitive advantage: Toward a conceptual integration. Journal of Management, 18(1), 77–91.
Lado, A. A., & Wilson, M. C. (1994). Human resource systems and sustained competitive advantage: A competency-based perspective. Academy of Management Review, 19(4), 699–727.
Latour, B. (1987). Science in action. Cambridge, MA: Harvard University Press.
Latour, B. (1999). Pandora’s hope: Essays on the reality of science studies. Cambridge, MA: Harvard University Press.
Mahajan, V., Sharma, S. Bettis, R.A. (1988). The adoption of the M-Form organizational structure: A test of imitation hypothesis. Management Science, 34(10), 1188–1201.
Meyer, J. W. and B. Rowan (1991). Rationalized Organizations: Formal Structure as Myth and Ceremony in The New Institutionalism in Organizational Analysis Edited by T. C. Powell and P. J. DiMaggio. Chicago, University of Chicago Press.
Montes, F. J. L., Jover, A. V., & Fernandez, L.M.M.. (2003). Factors affecting the relationship between total quality management and organizational performance. The International Journal of Quality and Reliability Management, 20(2), 188.
Mottiwalla, I., & Fairfield–Sonn, J. (1998). Measuring the impact of expert systems. Journal of Business and Economic Studies.
Pennypacker, J., & Grant, K. (2003). Project management maturity: An industry benchmark. Project Management Journal, 34(1), 4–11.
Peteraf, M. (1993). The cornerstones of competitive advantage: a resource-based view. Strategic Management Journal, 14 (3), 179–91.
Philips, J. J. (1998). Measuring the return on investment in organization development. Organization Development Journal, 16(4), 29.
Philips, J. J. (2002). Measuring return on investment in government: Issues and providers. Public Personnel Management, 31(2), 225.
Phillips, P. P., & Philips, J. I. F. (2004). R0I in the public sector: Myths and realities. Public Personnel Management, 33(2), 139.
Powell, T. C. (1995). Total quality management as competitive advantage: A review and empirical study. Strategic Management Journal, 16, 15–37.
Reginato, J. & Ibbs, W. (2002). Project management as a core competency. PMI Research Conference, Seattle, WA.
Roos, J., Roos, G., Dragonetti, N.C. & Edvinsson, L (1997). Intellectual capital: Navigating in the new business landscape. Basingtoke, MacMillan.
Russ, R. (2001). Economic value added: Theory, evidence, a missing link. Review of Business, 22(1), 66–71.
Smith, S., & Barker, J. (1999). Benefit-cost ratio: Selection tool or trap? PM Network, 23–26.
Soderlund, J. (2004). On the evolution of project competence: Empirical regularities in four Swedish Firms. IRNOP VI. Torku, Finland.
Teece, D. J. (1980). The diffusion of an administrative innovation. Management Science, 26(5), 464–470.
Thomas, J., & Mullaly, M. (2005). What’s the benefit? Challenges in demonstrating the value of project management. PMI North American Global Congress. Toronto, ON, Canada.
Tsai, W. (2002). “Social Structure of Cooperation within a multi-unit organization: coordination, competition and intraorganisational knowledge sharing.” Organization Science 13(2): 179-190.
Voepel, S. C., Leibold, M., Eckoff, R.A. & Davenport, T.H. (2006). The tyranny of the balanced scorecard in the innovation economy. Journal of Intellectual Capital, 7(1), 32–58.
Webber, S., Simsarian, T., & Maria, T. (2004). Project managers doubling as client account executives. Academy of Management Executive, 18(1).
Wernerfelt, B. (1984). A resource-based view of the firm. Strategic Management Journal, 12, 75–94.
Westerveld, E. (2003). Project excellence model: Linking success criteria and success factors. International Journal of Project Management, 21, 411–418.
Westphal, J. D., Gulati, R., Shortell, S.M. (1996). The institutionalization of total quality management: The emergence of normative TQM adoption and the consequences for organizational legitimacy and performance. Proceedings of the Academy of Management_Conference in Vancouver, British Columbia.
Westphal, J. D., Gulati, R. & Shortell, S.M.. (1997). Customization or conformity? An institutional and network perspective on the content and consequences of TQM adoption. Administrative Science Quarterly, 42, 366–394.
Zbaracki, M. (1998). Rhetoric and reality of total quality management. Administrative Sciences Quarterly, 43(3), 602–636.
©2006 Project Management Institute
Are you future ready? Our Pulse of the Profession® research shows that organizations that prioritize maturing their delivery capabilities enjoy more successful outcomes.