Introduction
The objective of this paper is to describe new research leading to a procedure that can help managers measure their Return on Investment for Project Management (PM/RoI). Previous research investigated this subject, and though the results were not statistically significant, they were encouraging.The new research described in this paper will remedy that previous study’s shortcomings. Additionally, this new study addresses the impacts of characteristics of newer companies, like devolved power, a stronger emphasis on intellectual capital, powerful brand identification, and a premium on project-driven services, as well as well as the characteristics of older companies, like strong, centralized command-and-control management. This paper will look at a few select companies chosen for the study, and their PM practices will be studied in much deeper detail than those of the 38 companies and 17 projects of the previous study.
The paper does this by quantitatively examining current PM processes and practices of five or six different companies. Each of these companies or organizations submitted at least three projects for detailed investigation, which allowed more in-depth analysis. The PM Benchmarking procedure developed and used to assess the PM Process Maturity in the previous study has been upgraded to accommodate the integration of additional metrics for this study. The new PM Benchmarking Assessment Tool is based upon six project life-cycle phases and nine PMBOK® Guide functional areas. The Five-Level Berkeley Project Management Process Maturity Model introduced in the previous study has been improved and will be used in this study.
This information is used to measure the relative sophistication and maturity of different organizations and industries. Then actual cost, schedule, quality, and customer satisfaction information are collected from these organizations for recently completed, representative projects. The previous research has already shown a positive association to exist between a company’s PM Maturity and its project cost and schedule performance. In this new study, cost and schedule data are analyzed in more detail. For example, PM costs are more clearly delineated.The new study also addresses the complication of budget and baseline consistencies between companies (e.g., authorized budget at concept versus authorized budget at contract signing).
These data are used, in conjunction with data from the past study, to develop curves that show predicted cost, schedule, quality, and customer satisfaction performance levels for a specific PM Maturity level. These curves can then be used by the individual organization to estimate what project performance gains might reasonably be expected if the organization takes steps to improve its PM Maturity.That information, in turn, can be used to estimate its specific PM/RoI.
Previous Research
Previous work in this arena includes the work of Ibbs and Kwak (1997). This study developed and presented the five-step Berkeley PM Process Maturity Model to better understand and locate an organization’s current PM process level. The novel feature is that it incorporates a learning component, which many companies purport to support but in reality do not. The Berkeley Model is an adaptation from Crosby (1979), the Software Engineering Institute (1993) and McCauley (1993).
The next step was to use this model to benchmark 38 companies in four different industries. The express purpose was to test the research hypothesis:
Ho: Project Management has Value to Companies that can be quantitatively demonstrated.
Though the research broke new ground, it did not prove this hypothesis to a statistically significant standard. Namely, the Berkeley team found that:
• There was an association, but not a statistically significant relationship, between an organization’s PM Maturity and its ability to execute projects more effectively.
• There was an association, but not significant, between a company’s PM expenditures and its project effectiveness.
• There was no meaningful relationship between the number of professionally-trained project managers (including PMPs) and the company’s ability to execute projects.
• Companies were seen to be relatively weak in the Risk Management knowledge area and in the Project Execution and Project Support phase areas, which confirmed the suspicions of many industry professionals.
Exhibit 1. Berkeley PM Process Maturity Model
This Phase 1 study had weaknesses though. Among these weaknesses are:
• The survey population was self-selecting and the PM Maturity assessment was conducted principally by the companies themselves. Our limited research funding precluded more comprehensive quality control.
• PM Maturity was contrasted with Project Cost and Schedule Performance Indices. One complication is the question of budget and baseline consistencies between companies (e.g., authorized budget at concept vs. authorized budget at contract signing. For instance, how are change orders handled? Which baseline budget should be used for reference purposes?).
• Perhaps more importantly, the Cost Performance Index is probably of less importance than the Schedule Performance Index for those enterprises whose revenues are derived from selling the project’s deliverable. For instance, the costs of developing software are not tightly coupled to their sales price.What is of more concern to a software vendor is time-to-market and first-market-mover status.Measuring the relationship between those downstream profits and PM effort is difficult.
• PM cost accounting structures differ radically among companies. Companies amortize expenditures on PM training, for instance, quite differently.Moreover, the line between operations and PM is rarely clearly delineated, so salaries, for one, are quite difficult to apportion. On a more global level, many corporate accounting systems are designed to track function (marketing, engineering) but not the portion of engineering or accounting that directly, let alone indirectly, supports a project.
• The value of PM was measured by its RoI. Some have argued for different measures. For instance, Knutson (1999) presents a benefit- cost calculation, but ignores the time value of money and subtleties such as how to amortize project-related capital costs. Crawford and Pennypacker (2000), on the other hand, has proposed a Balanced Scorecard approach, which still does not address numerous details like amortizing capital costs associated with PM.It also puts an artificial and masking weight on qualitative aspects in an attempt to balance them with quantitative factors.
• Other investigators (Hartman & Skulmolski 1999; Gareis & Huemann 1999, for example) have reiterated the need to determine the PM RoI. They have, however, not defined specific steps to quantify and measure this entity.
Berkeley PM Process Maturity Model
The five-step Berkeley PM-Process Maturity Model is used to establish an organization’s current PM maturity level. This model demonstrates sequential steps that map an organization’s incremental improvement of its PM processes.
As previously stated, the five-step Berkeley PM Process Maturity Model was updated from previous research. The new model incorporates Management at the Project Level for Level 3, Management at the Corporate Level for Level 4, and Continuous Learning for Level 5 (see Exhibit 1).
The model progresses from functionally driven organizational practices to project-driven organizations that incorporate continued PM learning.An organization’s position within the model signals their position relative to the other organizations in their industry class or otherwise that have been assessed.
Level 1: Ad-Hoc
At the Ad-Hoc Stage, there are no formal procedures or plans to execute the project. The project activities are poorly defined and cost estimates are inferior. PM-related data collection and analyses are not conducted in a systematic manner.Processes are unpredictable and poorly controlled. There are no formal steps or guidelines to ensure PM processes and practices. As a result, utilization of PM tools and techniques is extremely inconsistent and applied irregularly, if at all, even though the individual project manager may be very complacent (from Ibbs 2000).
Exhibit 2. Benchmarked PM Phases and Knowledge Areas (Ibbs 2000)
Level 2: Planned
At the Planned Level, informal and incomplete processes are used to manage a project. Some of the PM problems are identified, but these problems are not documented or corrected. PM-related data collection and analysis are informally conducted but not documented. PM processes are partially recognized and controlled by project managers. Nevertheless, planning and management of projects depends largely on individuals.
An organization at the Level 2 is more team oriented than at Level 1. The project team understands the project’s basic commitments. This organization possesses strength in doing similar and repeatable work. However, when the organization is presented with new or unfamiliar projects, it confronts major chaos in managing and controlling the project. Level 2 PM processes are efficient for individual project planning, but not for controlling the project, let alone any portfolio of projects (from Ibbs 97).
Level 3: Managed at the Project Level
At Level 3, PM exhibits systematic planning and control systems that are implemented for individual projects. PM processes become more robust and demonstrate both systematic planning and control characteristics. The PM team typically works together in an informal setting. For the purposes of project control, most of the challenges regarding PM are identified and informally documented for each project.Various types of analyzed trend data are shared by the project team to help it work together as an integrated unit throughout the duration of the project. This type of organization works hard to integrate cross-functional teams to form a project team.
Level 4: Managed at the Corporate Level
For projects managed at the Corporate Level, PM processes are formal, while information and processes are documented informally. The Level 4 organization is fully integrated; it can plan, manage, and control multiple projects efficiently across an organization’s project portfolio. A PM process model is well defined, with project requirement systems in place but that are not regularly used. Project related data and records are formally and systematically collected, reviewed and distributed to the appropriate parties, but are not formally organized.Also, data are collected and analyzed to anticipate and prevent adverse productivity and quality impacts or other trends detrimental to project success. This allows an organization to establish a foundation for fact-based decision-making.
In addition to effectively conducting multiple project planning and control for multiple projects, the organization exhibits a strong sense of teamwork within each project and across projects. PM training is fully available when needed and is provided throughout the entire organization, according to the respective role of project team members.
Level 5: Learning
The key characteristic of companies that operate at the Learning Stage is that they continuously improve their PM processes and practices. Each project team member spends considerable effort to maintain and sustain the project-driven environment.Training is formally available when needed, presenting lesson learned and other techniques to improve PM on an ongoing basis. Project team members are typically together throughout the entire project duration and their individual roles are defined based by their strengths and experience. Problems associated with applying PM are fully understood and addressed on an ongoing basis to ensure project success. PM data are collected automatically to benchmark PM strengths and identify the weakest process elements. These data are then rigorously analyzed and evaluated to select and improve the PM processes. Innovative ideas are also vigorously pursued, tested, and organized to improve processes.
Exhibit 3. Performance Definitions
Formal comprehensive requirement systems exist and are used regularly.A PM process model is formally defined, distributed and discussed by all of the team members using previous project experience as a guideline. Additionally, a PM consulting group is chartered and communicated throughout the organization.
Each level within the Berkeley PM Process Maturity Model includes an assessment of PM processes and practices based upon six PM processes and nine knowledge areas (see Exhibit 2). By assessing each organization along these boundaries, PM strengths are weaknesses are determined. This allows companies to prudently invest in areas to improve upon their weaknesses.
PM Maturity Assessment Tool
Value, from our perspective, is the value provided to the shareholders of the enterprise. In order to derive the quantitative value of PM, the costs associated with PM must be delineated. As previously stated, accounting measures differ radically among companies when applied to collecting PM costs. Therefore, many companies experience difficulties when they try to account for all PM-related costs.To combat this, we have added an extensive project cost category collection section to the Assessment Tool that collects the following data:
• Direct labor and direct labor burden costs for PM personnel
• Direct labor and direct labor burden costs for PM support personnel
• Costs of hardware, software, and communications systems used for PM
• Costs associated with training, travel and per diem for PM personnel
• Costs for consultants and subconsultants directly supporting projects
• Costs of building rents and/or mortgages for PM personnel offices.
All of the costs collected will be amortized over a regular schedule to determine the discounted value of the expenditures. By using the discounted amounts, the time-cost of money can be accounted for.
In addition to collecting specific PM costs, the Assessment Tool also collects specific project cost and schedule data. For each project, the Cost Performance Index and Schedule Performance are calculated from the equations in Exhibit 3.
In this study, a project finishing under budget or ahead of schedule would have a CPI or SPI > 1. These budget and duration values should be adjusted to reflect any change orders that project clients authorized. It is also necessary to note that our use of CPI and SPI differs from the PMI® definitions of cost and schedule performance indices (PMBOK® Guide 1996). The PMBOK® Guide defines CPI at the ratio of budgeted cost of work performed to the actual cost of work performed, and defines SPI as the ratio of budgeted cost of work performed to the budgeted cost of work scheduled, both used at a given time during the project lifecycle to compute earned value.We, on the other hand, are interested only at evaluating cost and schedule performance once the project is complete.
Because there are other perspectives as to what measures define value, we have also included in the Assessment Tool a section to collect Key Performance Indicator (KPI) metrics.Typical KPIs involve quality standards (defect rates, percentage of product returned/rejected by the customer, etc.), customer satisfaction measures (percentage of original customer performance requirements met), and project management response measures (ability to process owner-requested changes, time to process change orders, etc.). The Assessment Tool allows individual companies to report the KPIs that they use internally to determine project success and include them in the determination of PM value.
Benchmarking Analysis
In the original study, a three-part PM benchmarking tool was used to effectively and efficiently collect information about a company’s general organizational profile, its PM Maturity level, and cost and schedule performance from previous representative projects. With the aforementioned improvements to the Berkeley Maturity Model and Assessment Tool, we can better define an organization’s PM Maturity, which will yield more a more accurate PM/RoI calculation.
The first study selected four different industries and application areas to collect PM information (Engineering-Construction, Information Management and Movement/Telecommunications, Information Systems/Software Development, and High-Tech Manufacturing—the new study will allow for additional industries and application areas, including all PMI Specific Interest Groups). Organizations were further defined by size and number of years of experience in PM practices. It is important that organizations be segregated by the above definitions so that there can be a comparative study of PM practices across industries and among peer organizations within one industry. Peer organizations are compared on the basis of:
• Their performance vs. the average, best and worst of class for the nine PMBOK® Knowledge Areas
• Their performance vs. the average, best and worst of class for the six different Phases
• Their average PM cost vs. the average, best and worst of class
• Their PM Maturity vs. the Project Cost Performance Index and Schedule Performance Index curves in our database.
These comparisons, along with the PM/RoI calculation, allow organizations to determine the right amount of PM for their businesses and to identify how to improve their PM practices and what the benefit of such improvements will be.Additionally, organizations can determine where other companies are putting their PM emphasis and why.
Summary and Next Steps
This paper is intended to report the progress the second PMIsponsored research study for measuring Project Management’s value by the Berkeley PM/RoI team. Previous research has already shown a positive association to exist between a company’s PM Maturity and its ability to effectively execute projects. The improved the Assessment Tool can collect data for the six project life-cycle phases and nine PMBOK® functional areas, as well as serve as a vehicle to collect detailed cost, quality and customer service information. This new data, collected in great detail from five to six assessment participants, will dovetail with existing cost and schedule data from the previous study’s assessments. The results will be a better definition of curves that show predicted cost and schedule performance levels for defined PM Maturity levels. These curves can then be used by senior officers of individual organizations to estimate what project performance gains might reasonably be expected if the organization takes steps to improve its PM Maturity. That information, in turn, can be used to estimate its specific PM/RoI, a process defined by Ibbs and Kwak (2000).
Additionally, PM Maturity will be compared with additional Key Performance Indicators to further enable managers to deduce the value of PM. The Assessment Tool will allow us to test for statistical correlations between variables such as:
• PM Maturity and project results
• PM Maturity and low project result variance
• PM Maturity and company expenditures on PM.
If a positive correlation exists, with the above variables, then we can state that PM has a quantifiable value to organizations.
This research project is scheduled to finish in Fall 2001. Progress and final results will be published in PMI forums as the research moves forward.
