Play the numbers game

Abstract

Tired of justifying your value? Let numbers do it for you! If you have grown weary of always justifying the value of project management, this session is for you. Learn how to develop a scorecard to track ongoing project management performance. By doing so, you will also be able to measure the business impact of project management to your organization. You’ll learn the basics of setting up a measurement program designed to help improve your project management performance as well as justify your organization’s continued investment in project management.

This one-hour session offers you a comprehensive introduction to measurement. You’ll learn how to assess what’s performing well and what needs improvement, to build your business case for project management improvement initiatives. Become proactive in implementing measurement strategies aimed at improving your organization’s project management performance.

Introduction

Why measure performance? Measuring project management performance, and ultimately value, will help organizations achieve one or more of the following goals:

• Identify the business impact of implementing project management improvement initiatives
• Compare costs to benefits of project management improvement initiatives
• Determine if a project management improvement initiative is accomplishing its objectives
• Assist in marketing future project management improvement initiatives.

When you measure performance, you are gathering information to help you make management decisions to effect change that, hopefully, will improve that specific performance. These measures are a starting point to help measure value. Value measures are cumulative and provide information on the performance of the organization rather that the performance of a project. In measuring value, you are trying to demonstrate that decisions you made to implement change have (or have not) added value to the organization. The goal of this paper is to provide you with a systematic approach to performance improvement. This measurement approach is an ongoing process of establishing metrics; collecting, analyzing, reviewing, and reporting performance data; using that data to drive performance improvement; and using lessons learned to continually improve your performance measurement program.

Measurement Background and Overview

Large organizations collect millions of data points every day, but few are able to use these measurements effectively. The challenge is to make measurement everyone’s job and to use the results to improve social and organizational performance. The current attitude toward measurement in most organizations is negative. This is mostly due to the judgment that follows, where only a few are rewarded, and most are punished as a result of the measurement. Ninety-three percent of organizational leaders believe that measurement is important in influencing business outcomes, only 51% are satisfied with their current systems, and only 15% are very satisfied. Even accountants don’t like them: a recent study showed that only 35% of respondents rated their organization’s performance measurement systems as effective or very effective (Spitzer, 2007.) We need to change those statistics. We need to implement measurement systems that measure and drive organizational success.

Why Measure Performance and Value?

One reason we measure is to succeed. Organizations can use performance measurement to manage strategy, systems, and processes more effectively and more consistently, which creates a tremendous competitive advantage for them. To do so, you first need to identify areas for performance improvement, then benchmark against the industry to be able to set reasonable targets for improvements. As your reporting starts to demonstrate the change in performance, you’ll want to evaluate the effectiveness of the change. If the change in performance is a positive change that is occurring on a repeating basis, on more than one project, then we are ready to measure the value to the organization. Remember, performance measurement measures “changes” — changes that improve performance. An example of this is measuring the effect of mandating that project managers document the scope of a project and have change orders executed only after they are approved by the project sponsor. Assume this is a new “process” and the result of this change has resulted in projects realizing a 30% improvement in schedule and financial performance. These measures had to be collected fairly often, probably weekly, and lasted for the duration of the projects.

In value measurement, you are trying to demonstrate that the decisions you made to implement the change — in this case, requiring scope and change documents — have added value to the organization. Usually, improved performance can be translated into value. For example, improving schedule performance for all of your projects over a period of a year can be translated into improvement in average project cycle time, which can be translated into improvement in time to market, which can add significant value to your organization. Value measures provide information on the performance of the organization rather than the performance of a project. These measures must be collected over a longer period of time and over your portfolio of projects.

There is a broad list of possible measures. There is no single set of measures that universally applies to all companies. The appropriate set of measures depends on the organization’s strategy, technology, and the particular industry and environment in which they compete. The top 10 recommended measures that an organization should benchmark to lead to project management success (Pennypacker, 2005) are:

1. Return on Investment ((Net Benefits/Costs) x 100) – The most appropriate formula for evaluating project investment is net benefits divided by cost. By multiplying this result by 100, this calculation determines the percentage return for every dollar you’ve invested.
2. Productivity – Productivity is output produced per unit of input.
3. Cost of Quality – Cost of quality is the amount of money a business loses because its product was not done right in the first place.
4. Cost Performance –The cost performance index is a measure of cost efficiency. It’s determined by dividing the value of the work actually performed (the earned value) by the actual costs that it took to accomplish the earned value.
5. Schedule Performance (Earned Value/Planned Value) – The schedule performance index is the ratio of total original authorized duration versus total final project duration. The ability to accurately forecast schedule helps meet time-to-market windows.
6. Customer Satisfaction (Scale of 1–100) – Customer satisfaction means that customer expectations are met. This requires a combination of conformance to requirements and fitness for use. The customer satisfaction index comprises hard measures of customer buying/use behavior and soft measures of customer opinions or feelings. (Repeat and lost customers/revenue from existing customers, complaint returns.)
7. Cycle Time – There are two types of cycle time: project cycle and process cycle. The project life cycle defines the beginning and the end of a project. Cycle time is the time it takes to complete the project life cycle. Cycle-time measures are based on standard performance. That is, cycle times for similar types of projects can be benchmarked to determine a Standard Project Life Cycle Time.
8. Requirement Performance – Meeting requirements is one of the key success factors for project management. To measure this factor you need to develop measures of fit, which means the solution completely satisfies the requirement.
9. Employee Satisfaction – An employee satisfaction index will give you one number to look at to determine employee morale levels. The ESI comprises a mix of soft and hard measures that are each assigned a weight based on their importance as a predictor of employee satisfaction levels.
10. Alignment to Strategic Business Goals – Most project management metrics benchmark the efficiency of project management — doing projects right. You also need a metric to determine whether or not you’re working on the right projects. This can be done by using a Likert scale from 1–10, distributed to an appropriate mix of project management professionals, business unit managers, and executives, to rate the statement, “Projects are aligned with the business’s strategic objectives.”

These metrics help support determining the value of project management to organizations. For instance, financial measures would be the value “category” for the results of the metrics used such as ROI, productivity, and cost savings. We need to capture and record the results of these measures to support whether or not changes made to processes or systems actually did prove to be valuable to the organization as a whole.

Why is all of this important? Well, in one instance, the National Research Council (NRC) completed a planned three-year review and assessment of the U.S. Department of Energy’s project management. In the final assessment report, the Committee for Oversight and Assessment of the U.S. Department of Energy made the following observation:

We can’t improve what we don’t know; but we also can’t improve by measuring the wrong things. The measurement “hall of shame” is full of organizations that do a good job of producing “results” on the wrong measures of success. For example:

• A hotel chain that responded to a cash flow crisis by mercilessly cutting costs when it should have been focusing on occupancy rate. This resulted in short-term cash, but a further deterioration of the business.
• A retail store chain that wanted to increase customer satisfaction using a “smile index” to measure employees on the extent to which they smiled at customers. Customers knew the smiles were phony, and the morale of employees, forced into unnatural acts, plummeted.
• Call center measures are most commonly used to drive quick response and low costs, rather than customer insight and gross sales to grow the business. Ironically, by the existing measurement system, the faster they hang up on customers, the better their performance! (Spritzer, 2007).

There are typically barriers in organizations when trying to set up and implement measurement programs. One of the greatest is a lack of understanding: understanding the strategies and objectives of the organization, understanding the key business processes, and understanding stakeholder needs. Other barriers include senior management involvement, employee involvement, accountability for measures, lack of a framework to measure, lack of an effective communication plan, resource shortages, and an overall lack of a sense of urgency.

A Project Management Measurement Framework

There are six steps needed to establish and maintain an effective project management measurement system. They constitute a systematic approach to performance improvement through an ongoing process of establishing metrics; collecting, analyzing, reviewing, and reporting performance data; using that data to drive performance improvement; and using lessons learned to continuously improve the process. The six steps that comprise this project management framework (PEMARI) are (1) Measurement Planning, (2) Establishing and Updating Measures, (3) Measuring Performance, (4) Analyzing Data, (5) Performance Reporting, and (6) Continuous Improvement (see Exhibit 1).

Exhibit 1 – PM Measurement Framework (PEMARI)

Measurement Planning

In the Planning phase of the PEMARI, identify your measurement initiative team. There is generally a sponsor, a project manager, representatives from stakeholders, and team members. Once you identify the team, assign roles and responsibilities. Following this step, you establish a clear understanding of terminology so that the team is clear and everyone is aware of the process you will be following. The next step, and this is one of the most important steps, is to discuss and set your goals and objectives. The goals could be organizational goals to reduce costs, improve quality, schedule performance, and so forth. These goals should align with your project management goals and objectives (more predictable project performance), and your measurement program goals and objectives (cost/benefit of improvement initiatives). Always be clear on what you are doing and why you are doing it.

In the planning phase, make sure there are no other measurement initiatives going on. Don’t duplicate efforts of another department or unit. Once you are sure you are not, begin to define your objectives and the scope of the program. Develop high-level schedule of activities and resources, develop individual project plans, and then choose your implementation approach.

Establishing and Updating Measures

The first thing to do at this stage is to develop a list of potential measures. These should be developed collaboratively with the measurement team. Ensure that you have criteria for good measures. These might include that the measure provides meaningful information, can be supported by valid and available data, and is cost-effective to capture (an important criteria for most organizations). The data will need to be acceptable to the stakeholders, be repeatable, actionable, and align with organizational objectives. Good measures align with organizational objectives (see Exhibit 2).

Exhibit 2 – Aligning Measures with Strategic Objectives

Another important step in this phase is to prioritize and select a few critical measures based on the agreed-upon criteria. Keep the number of measures to no more than six. The four most important criteria are importance, data availability, “actionability” and alignment to organizational objectives.

Now you are ready to develop a scorecard of vital measures. You’ve selected your measures, have developed measure packages for each measure, answering “What, Why, When, Who, and How” and established your performance goals, with your targets, baseline, current results, and the variance. Review your scorecard periodically for relevance and effectiveness (see Exhibit 3).

Exhibit 3 – Performance Scorecard

Through this process, you will be collecting data for analysis. Data collection should be matched to the need for decisions and to the cost of collection and processing. You always want to ensure data quality. Data needs to be focused, flexible, simple and aligned, and consistent to allow for comparisons.

Analyzing Data

Use your scorecard to analyze your data. Analyze and validate your results. Remember why you are doing this, collect and organize the data to answer that question. Present the data in a way that clearly communicates answers to the questions. Also ask questions like, “How does the actual performance compare to the goal or standard?”, “If there is a significant variance, is corrective action necessary?”, and “Are new goals or measures needed?” Use the data in a positive way.

Performance Reporting

Developing a communications plan is critical. Consider who your target audience is, clearly state why you performed this measurement, and then decide how and when you’ll want to communicate the results. You definitely need to share the results with your stakeholders. The goal is for the audience to understand what they are receiving, realize what actions are needed, take action, and follow up on the impacts of those actions.

Continuous Improvement

The last step is review for factors that impact program, including your barriers to success, who comprised the measurement team, and whether you met organizational strategies and objectives. Continually monitor the measurement team and keep track of the organizational strategies and objectives, ensuring that the measures are still relevant and aligned. There are organizations that have data on best practices for specific industries — search them out. And finally, learn from the feedback received. Make recommendations to change the process if warranted and remove items that hinder the process.

In summary, in a performance metric process you should:

• Manage your stakeholders’ expectations
• Get senior management buy-in
• Choose the right measures, understanding the maturity level of your organization
• Create a sense of urgency
• Pilot and implement in phases
• Ensure that the measurement program delivers value.