Justifying the value of a project management deployment in your organization through an ROI impact study

Abstract

Project management has grown in its use over the last several years. With 76% of IT projects failing (Crawford, 2002, p. 19), project management must be utilized to ensure project success. The Guide to the Project Management Body of Knowledge (PMBOK^® Guide) identifies a proven methodology that often is not utilized because it is not quantified to the value it returns to the organization. The purpose of this paper is to learn a methodology that establishes a credible approach to reporting return on investment (ROI). The ability to report the ROI of a project management initiative is necessary in order to sustain support and funding for continued project management initiatives. You will learn how ROI can be reported through a 10-step approach to collecting and reporting data. From this paper you will see the potential for implementing a reporting system that will be embraced by all senior level management.

In this paper you learn a methodology that follows the initial introduction of the project management methodology through the completion of a project. You will see how to report the financial and the intangible benefits from project management. This will allow you to build a balanced scorecard reporting system that shows the value of project management in your organization.

Introduction

The capability to report the value of any expenditure in an organization is necessary in order to sustain funding and support. Organizations must spend efficiently and effectively to maintain profitability. In order to determine which programs generate revenue, they must be evaluated to determine their value.

Project management can create many improvements, but unless they are measured, you cannot justify increased or further investment into these efforts. Investments are required to be made in software, systems, human resources, physical resources, training, and time. When all totaled, the investment in project management can be quite large. Sooner or later the chief financial officer (CFO) will challenge what the organization is getting for their investment in project management.

Identifying what to measure and how to measure it can be a challenge. The Phillips ROI methodology is an extensively used and accepted method to evaluate program performance. The Phillips ROI process provides a common methodology that is replicable and can be validated across an organization to ensure acceptance throughout the organization.

The Phillips ROI methodology was developed by Jack Phillips, based on Kirkpatrick's four levels of training evaluation. Phillips developed this methodology to tie evaluation to the organization's strategic goals. Project management initiatives can be evaluated—beginning with the initial commitment, through training and methodology, through project completion—to assess the return on the investment.

Return on Investment

Project management has grown in popularity over the last several years. This is possibly due to data such as Phillips (2002) quotes the Standish Group as reporting on IT projects:

• 31% of projects are cancelled before completion
• 88% are past deadline, over budget or both
• For every 100 starts, there are 94 restarts
• Average cost overrun is 189%
• Average time overrun is 222%.

Investments in project management can be a large capital investment for many organizations. Costs can include personnel, training, software, consulting, computer systems, and the purchase of methodologies. The benefits realized from a project management implementation can include reduced schedule delays and cost overruns, and increased customer satisfaction. ROI is a methodology that we look to for data collection, analysis, and reporting of the findings.

The ROI methodology that this presentation will present contains five levels of evaluation and six types of data. The five levels are:

• Level 1—Reaction, satisfaction, and planned action
• Level 2—Learning
• Level 3—Application
• Level 4—Impact
• Level 5—ROI

These five levels are also referred to as types of data. There is a sixth type of data that is reported, classified as intangibles. Intangible data is not converted to financial data, because it can be accepted as is, or it is accepted that collection and/or converting this data can take too long. The financial impact of converting the data from intangible data would be greater than necessary to report the findings that we have.

Exhibit 1—The ROI process

The Phillip's ROI process is a 10-step method. The model allows for a duplicative and credible method for collecting and reporting ROI. Project management requires an investment of human and capital resources to implement in any organization. Because senior-level support for project management can wane over time, ROI can help ensure that support will be sustained. The secondary purpose for using ROI methodology is that we maximize the utilization of project management. If the project management initiative is not implemented properly, it is possible that its projected success is reduced, as well as support for it. The hypothesis here is that if the value of the project management initiative is maximized and known, support will continue.

Evaluation Planning

Step 1—Develop Objectives of the Solution

Expected Outcomes

There are many reasons for implementing project management and why we would need to evaluate them. Project management ROI evaluations should be used to determine the following:

• Benefits versus costs
• Strengths and weaknesses of initiative
• Establishing priorities
• Establishing expectations.

Objectives should be identified for expected outcomes at each level of evaluation. The objectives are specified for each level:

• Level 1—Reaction, satisfaction, and planned action. Objectives should state the expected satisfaction with a project management initiative. An example could be that 100% of all employees have a positive feeling about how project management improves project execution.
• Level 2—Learning. Objectives should define the level of project management competency. An example could be that students must score 80% on project management tests.
• Level 3—Application. Objectives identify how, where, and when project management is used. An example could be that 80% of employees properly follow the project management methodology on projects.
• Level 4—Impact. Objectives identify the expected benefits from project management. An example could be that project delays will be reduced from 30% to 20% in one year.
• Level 5—ROI. Objectives should state what an acceptable or desired ROI is. An example could be that a 30% ROI will be realized after one year of project management implementation.

Step 2 - Develop Evaluation Plans and Baseline Data

Key to a successful evaluation will be planning how the evaluation will be conducted. Three key plans need to be developed in order to start your evaluation:

• Data collection plan—Identify where and what data you will collect.
• ROI analysis plan—Identify how you will convert the data to monetary value and identify where and who gets what credit.
• Project plan—An ROI evaluation should be treated like any other project and have a project plan.

Data Collection

Step 3—Collecting Data During the Solution

Level 1: Reaction, Satisfaction, and Planned Action

The success of an initiative such as a project management implementation requires commitment from the top down and the bottom up. A top-down commitment as shown by support from senior level management is necessary for a successful rollout. The executive-level officer needs to sponsor the effort with their clout and funding. They must understand the project management initiative so that they can effectively support it. Any prudent executive is sure to ask at some point what the organization is getting for its investment. A bottom-up commitment requires the workers to execute the initiative. To do this, they need to understand what to do. This is when the first level of data in the ROI process, “reaction, satisfaction, and planned action, ” is assessed. We all have heard the importance of “starting off on the right foot”; this refers to the beginning, when the attitude can be formed that can make or break the project management initiative (or any initiative that we evaluate). A positive reaction can make all the difference in the success of a project management implementation.

Phillips and Phillips (2007) identified some measurable reactions to the project:

• The project is relevant to my job
• The project is necessary
• I intend to implement the project
• The project is important to my success

There are several sources that you can use to collect this data:

• Internal and external customers
• Executives
• Project managers
• Functional managers
• Team members
• Project sponsors

There are several methods that you can use to gather this data:

• Questionnaires
• Surveys
• Interviews
• Focus groups

Level 2: Learning

Learning metrics are classic benchmarks used in the training profession; however, they tell much more of a story for the organization as any change is implemented. Change success depends on the ability of the organization's resources to understand the skills and knowledge needed to implement the change. We are comfortable doing what we understand and know. Here lies the importance of communicating the knowledge and skills needed for any initiative. The ability to successfully use project management is dependent upon the knowledge of the executors. Gaps in knowledge should be identified, so that they can be addressed to optimize the benefits of a project management implementation.

Metrics should measure:

• Skills
• Knowledge
• Attitude

There are several techniques that you can use to gather this data:

• Questionnaires
• Performance tests
• Performance observations
• Assessments

Step 4—Collecting Data after the Solution

Level 3: Application

The best technology, concepts, training, or intentions will have no impact if they are not used. We have seen the best training provided to no avail because it was not used when the student returned to the job. Business continued as usual with no use of the skills and knowledge obtained in the training. While the initiative is a hot topic in the organization the training will be supported, but as soon as budget cuts are necessary and the initiative has cooled, it gets cut. If the value the program returns to the organization is known, it is not likely that the program will get cut. Whether as an executive or a project manager, we must know the value of project management, so that we can get the required support.

There are many methods that you can use to identify whether project management is used. They should all be related to performance. First, the competencies needed to perform project management should be identified. A sample of tasks that project managers should be competent in performing are as follows:

1.     Determine project goals

2.     Determine product or serviceable deliverables

3.     Determine project management process outputs

4.     Document project constraints

5.     Document assumptions

6.     Define the project strategy

7.     Identify performance criteria

8.     Determine key resource requirements

9.     Define an appropriate project budget and schedule

10.   Refine project requirements

11.   Create the work breakdown structure

12.   Develop the resource management plan

13.   Refine project time and cost

14.   Establish project control

15.   Develop a formal and comprehensive project plan

16.   Obtain project plan approval

17.   Commit project resources

18.   Implement the project plan

19.   Manage project progress

20.   Communicate project progress

21.   Implement quality assurance reports

22.   Measure project performance

23.   Refine control limits

24.   Take timely corrective actions

25.   Evaluate the effectiveness of corrective actions

26.   Ensure compliance with the change management plan

27.   Reassess project control plans

28.   Respond to risk event triggers

29.   Monitor project activity

30.   Obtain final acceptance of deliverables

31.   Document lessons learned

32.   Facilitate administrative and financial closure

33.   Preserve project records

34.   Release project resources

For each of these tasks you should identify the skills and knowledge required to perform competently. Once this is known, you can use surveys, questionnaires, and observations to determine whether the competencies are present.

Action Plans

The action plan is useful to identify what skills and knowledge learned or identified will be used. The action plan should be developed during the implementation of the project management initiative. In most cases, this is done during training. This can be developed by the individual or by their management. This can be seen as a performance standard. The action plan helps take the training from the classroom and put it into use on the job. If planned together with the individual and the manager, the action plan can become a performance contract. The action plan will identify what will be done and when.

Level 4: Impact

The levels of data already identified establish the data trail to validate the ROI impact study. The data leads us to level 4, business impact. Once we can determine the business impact, we can convert the data to monetary values. There are two types of data that we will look to collect; hard data and soft data.

Hard data is more tangible than soft data. Look to the triple constraint (scope, cost, and time) to identify data that you can collect. Categorize hard data into four types:

• Output
• Time
• Cost
• Quality

Soft data is more difficult to convert to monetary value. It can be converted if desired, but soft data can also be reported as intangibles. We will discuss intangible data later in this paper. Some categories of soft data could be:

• Work habits
• Work climate/satisfaction
• Customer service
• Employee deployment
• Initiative and innovation

Data Analysis

Step 5—Isolate the Effects of the Solution

Isolating the effects of the solution is essential to establish the credibility of the solution. There can be multiple reasons that an improvement occurs, such as a change in the economy, or other factors. This thought and opinion will occur throughout the organization's leadership. Isolating these factors and reporting the impact of each will add credibility to the impact that you report from the solution you implemented.

In order to identify the influences that could have effected the change that occurs you need to go to multiple sources. These can include the following:

• The project sponsor
• The project team
• Project stakeholders
• Subject matter experts
• Middle and top management

Techniques and tools that can be used to isolate the impact that project management has include:

• Control groups
• Trend line analysis
• Forecasting methods
• Estimates
• Participant's estimates of the impact
• Management's estimates of the impact
• Internal or external expert impact

Example of a Reduction in Schedule Delays

Use Exhibit 2 to illustrate the use of participant's estimates of the impact. In the exhibit you multiply the impact by the confidence level. Then multiply the resulting value by the value of schedule delay reduction. To illustrate this, consider the following example. For every day of delay that is reduced, $100K was saved. Historically, there is an average of 25 days that the planned schedule is missed by. After a project management initiative is implemented, there is an average of 15 days that we miss the planned schedule by. The project management initiative has resulted in a savings of 10 days, valued at $100K for each day. The result is $1 million dollars saved. Project management has an adjusted impact (after considering confidence levels) of 43% (52% x 83% = 43%). To find the isolated impacts, multiply the adjusted impact level by the value realized in savings; in this example, we realized $1 million. This results in project management having a $430,000 isolated impact on the $1 million savings. When determining the ROI for the project management solution, $430,000 will be used as the benefit, not $1 million. Understand that this will increase the credibility of the value that project management returns to the organization.

Exhibit 2—Measurement of impact of contributing factors

Step 6—Convert the Data to Monetary Value

It's time to show the money. Converting data to monetary value can be a challenging part of the process. Business results must be quantified to convey monetary value. This is necessary to determine ROI. ROI will allow you to show the value of project management.

Previously, we identified that there are hard costs and soft costs for all performance improvement efforts. The measures are categorized as output, costs, time, and quality. This should be straightforward to those in project management. They are in alignment with the triple constraint for project management, as shown in Exhibit 3.

Exhibit 3—The triple constraint

Finding standard values to convert can come from different departments throughout an organization:

• IT
• Finance and accounting
• Sales
• Marketing
• Operation
• Engineering
• HR
• Procurement

Baseline data can become extremely useful when standard values are not available. Look at previous projects. Identify from project records what they were predicted to accomplish and where they actually ended up. Statistical information can be identified for project performance. Are they historically late or over budget, and by what amount? This can be compared to project performance after a project management implementation.

Project management initiatives can include implementing all or some of the following:

• Project management office (PMO)
• Project management methodology
• Project management training
• Project management software and systems

As previously stated, converting data is challenging. To better understand data conversion, follow these five key steps to convert data to monetary values:

• Focus on a unit of measure
• Determine the value of each unit
• Calculate the change in performance data
• Determine the annual amount of change
• Calculate the annual value of the improvement

Step 7 - Identifying Intangible Benefits

There is another type of data that may have an equal or greater value to an organization, and that is intangible data. We can report collected data as intangible data if it is determined that it would be too long to convert the data to monetary value or too expensive to convert to monetary value. If intangible data has value in and of itself, this also justifies not converting the data to monetary value.

Phillips (2007) identified several common intangibles. Here are few key intangibles that we can see in project management implementations:

• Innovation and creativity
• Timeliness
• Intellectual capital
• Reputation
• Capacity
• Capability
• Customer satisfaction
• Employee job satisfaction

Step 8—Tabulate the Costs of the Solution

The next step is to account for and report all costs, direct and indirect. Costs should include all resources, internal and external to the project team, that were dedicated to the project. Costs should be identified as loaded costs, to include all salaries and benefits that are required to obtain needed resources. There are several categories to consider when determining costs. Below is a recommended list:

• Analysis costs
• Planning costs
• Implementation costs
• Maintenance costs
• Travel and expenses
• Salaries (use loaded rates)
• Software costs
• Equipment costs
• Consultants
• Training

Step 9 - Calculate the Return on the Investment

Level 5: ROI

The final step before you report your findings is to calculate the benefit/costs ratio (BCR) and the return on investment (ROI). A recommendation to increase the credibility of your report is to stay conservative on your findings. When identifying costs and benefits you probably will have a range that you can report. For example, when reporting costs, if the range is between $200,000 and $250,000 report the $250,000 as the project costs. And when reporting benefits, if the range is between $900,000 and $1,000,000 report the $900,000 as the project benefits. Staying conservative will always result in your favor if the numbers that you report are challenged.

The calculations for BCR and ROI are:

BCR = Program Benefits ÷ Program Costs

ROI = (Net Program Benefits × Program Costs) × 100

ROI is reported as a percentage. Organizations should identify what they expect as an acceptable ROI. Should the project be a compliance project, ROI can be set at zero, or if the organization has 25% as their expected ROI level for other capital investments, set the ROI at 25%. This establishes an expected level of performance to measure to determine that we are meeting expectations.

Reporting

Step 10 - Generate the Impact Study

The impact study is your opportunity to tell “your” story. Communicate the report through meetings, intranets, weblogs, and so forth. The main objective is to communicate your findings in a way that can be understood. This will produce the support you need throughout your organization.

You need to show:

• That there was a plan
• That the plan was executed
• Where all the data came from
• How the data was interpreted

The impact study should have the following sections:

• General information
• Methodology for impact study
• Data analysis
• Costs
• Results
• Barriers and enablers
• Conclusions and recommendations
• Exhibits

Conclusion

We first acknowledged that project management requires a financial commitment from the organization. Resources, capital, and time will be committed. At some point, the organization will need to show the return on these investments. The Phillips ROI methodology is a credible methodology that allows you to collect and analyze the data and to report the ROI in a credible and accepted manner. The five levels of ROI allow you to see if you met the planned expectations for your project management initiative. This will maximize the use and probability of success for your continued project management efforts.