For a metric to be effective, data must be well defined, accurate and relevant. In a nutshell, a metric is only as reliable as the data that's fed into it. “Data is neither good nor bad, it's just numbers,” says Karl Wiegers, principal consultant at Process Impact, Clackamas, Ore., USA. “To make it useful, you need to interpret it, and to do that you need to establish definitions. If you and I aren't counting things the same way, we can't compare them.”
By establishing a baseline against which to measure project performance, a team can work toward process improvements on a percentage basis. Wiegers recommends tracking four categories of metrics over time: product size, duration, work effort distribution and defects.
Regarding work effort, project managers seldom compare how much of a team's time is spent on new software development compared with maintenance, says Wiegers. New software development creates the opportunity to do business and generate revenue, but managers can seldom make this assessment. What's more, these metric data help establish baselines, he says.
Creating a baseline against which to measure project performance gets tricky when you start measuring hard-to-pin-down metrics such as project value at very large companies, adds Steve Rollins, chief of project strategy at EPM Solutions Inc., Kansas City, Mo., USA. “If you find a way to cut four weeks off of scheduled time from 30 percent of the projects, that adds up,” says Rollins.
A project management office (PMO) often provides much needed oversight and consulting services to help a company gain more value from its projects. “If you can't see where you're going, how can you hit the ball?” says Rollins. “PMOs should be crucial departments at every company for measuring a project's ROI.”
However, defining data is only the first step. After a project manager has defined and gathered data to be used in a metric, it's just as important to use it in the right context. Data can mean different things to different people involved in a project, and each of those people will rate the project's success differently, depending on their role.
“The owner of a building construction project might be happy if it was completed on budget,” says Narayan Bodapati, chairman of the Department of Construction, School of Engineering at Southern Illinois University, Edwardsville, Ill., USA. “The construction manager wants safe working conditions because they pay workers' compensation benefits. Users have a different perspective on quality than owners and architects. Quality measures for a user might include the simple fact that the roof isn't leaking and the elevator is working.”
MEASURING SAFETY
Metrics ensure safety and reliability for Bechtel Hanford Inc., an environmental restoration unit of San Francisco, Calif., USA-based Bechtel Corp. The U.S. Department of Energy originally contracted Bechtel Hanford in 1994 to clean up nuclear waste left behind by the Manhattan Project. Today, the unit works to restore the 586-square-mile site in southeast Washington State, USA.
The 10-year operation, which involves 800 Bechtel and subcontracted employees, is divided into three major projects: decontamination and safe storage of reactors, remedial action and waste disposal, and surveillance and maintenance of inactive facilities. Project leaders have established a rigorous metrics program to keep actual cost, schedules, production and safety on track with work plans.
Vern Dronen, remedial action and waste disposal project manager, uses an earned-value system to measure project cost and scheduling. “We take the planned cost for work performed and compare that against actual cost for work performed to give us a metric at any stage of the project,” says Dronen.
For example, using budgeted and actual costs, Dronen can calculate the cost performance index (CPI). If budgeted cost for a work plan during a fiscal year is $4.17 million, and the actual cost is $4 million, the CPI is 0.959, and the project is coming in under budget. The CPI formula is:
Actual Cost of Work Performed (ACWP) / Budgeted Cost of Work Performed (BCWP)
Dronen compares budgeted cost for work scheduled to budgeted cost for work performed, and develops the schedule variance. The formula for determining schedule performance index (SPI) is:
Budgeted Cost of Work Scheduled (BCWS) / BCWP
A production metric measures actual tons of contaminated material disposed against planned tons of contaminated material disposed. At completion, Dronen's team will dispose of nearly 10 million tons of material, or 550,000 tons per year.
This project's safety metrics are based in part on injuries reported to the Bureau of Labor Statistics. Dronen also measures his project's safety record against truck accident rates furnished by the National Highway Traffic Safety Administration. So far, truck drivers have driven 6.3 million miles without an “at-fault” accident, far better than the national average. The national truck accident rate is 2.6 per 1 million miles traveled, according to the NHTSA.
Dronen monitors project progress by examining planned staffing compared to actual staffing. He also measures efficiency by looking at planned cost per ton of material disposed compared to the actual cost. “We typically report status on a monthly basis, but the project manager has access to data so he can manage on a weekly or daily basis,” he says. “These metrics allow a project manager to assess the welfare of the project on an almost real-time basis.”
A good project control system periodically measures projected and actual costs and time distribution to solve problems before they spiral out of control, says Bodapati. “You need predictive tools that will tell you whether something has changed that will increase projected cost or completion time,” he says. “Initially, there will be estimated cost, but you'll find out as you build that you're nowhere near that, and you can make adjustments.”
Homing Beacon
Compiling these baseline measurements requires large collections of data from similar projects, or a large data store. For example, the Energy Analysis program at the New York State Energy Research and Development Authority (NYSERDA), Albany, N.Y., USA, establishes norms by maintaining a set of energy statistics. The program's staff analyzes energy issues and publishes statistics and data as a source of objective information about all aspects of New York's energy production and use. Early in its data gathering process, officials determined that they weren't getting enough participation from the state's energy industry organizations, says Paul DeCotis, director of NYSERDA's Energy Analysis Program.
Users have a different perspective on quality than owners and architects. Quality measures for a user might include the simple fact that the roof isn't leaking and the elevator is working.
NARAYAN BODAPATI,
CHAIRMAN OF THE DEPARTMENT OF CONSTRUCTION,
SCHOOL OF ENGINEERING AT SOUTHERN ILLINOIS UNIVERSITY, EDWARDSVILLE, ILL., USA
“We found that paperwork was too onerous, financial incentives weren't high enough, and we were getting only 20 to 30 percent of what we expected,” says DeCotis. “So we lessened the paperwork and reduced the time required to contribute. We could still be rigorous and stringent but make it less work for the customer. We refined the program elements while still meeting our needs and doing the public good.”
Lack of a large pool of projects against which to measure timeliness, cost and work distribution historically has been a problem in the software industry, and this reference hole has made benchmarking difficult. Over the last several years, the International Software Benchmarking and Standards Group (ISBSG), Melbourne, Australia, has taken on the task of setting standards for custom-development projects. “You need to be able to say that you're on this platform, which is C++ or Java, and you're developing a portal application, how much will it cost and how long will it take to develop,” says Terry Wright, ISBSG president.
ISBSG's metrics and benchmarking program allows software developers to evaluate their project performance against other developers, more accurately estimate effort, time and cost, lower development risk, and build an experienced database of their organization's productivity.
For example, if a development team is contracted to build an accounting application that will run on UNIX and integrate with enterprise software such as SAP, says Wright, “people can bid on cost of a development project per function point, just like they bid on cost per square meter for the construction industry.”
You don't want to use data to reward or punish an individual. For example, if you're using metrics to evaluate code quality in a software development project, avoid using team members' names or singling out specific mistakes made by one person.
KARL WIEGERS,
PRINCIPAL CONSULTANT, PROCESS IMPACT,
CL ACKAMAS, ORE., USA
Measurement-Oriented Culture
Mounds of data and a litany of metrics are useless if a team isn't working toward the common goal of project completion.
“To be effective, a project manager has to be a social architect,” says Hans Thamhain, director of the project management program at Bentley College, Waltham, Mass., USA. “If you can involve a team member in a project in a way that they see themselves as part of the overall project team, and the project elements are agreed upon and they understand what steps they have to take, that provides positive reinforcement.”
As a team leader, a project manager should instill excitement about the task at hand: Tell a team member why they are on a project team and how the project fits into the company's plans. “You want people to be fully committed to the project,” says Thamhain. “If a team's product has passed a beta test, congratulate them. I've seen this give the team vibrancy. If you have that ambiance, it's a lot easier for people to work with management, and you'll have a team that has mutual respect for you as a manager.”
At the same time, metrics should be analyzed in a way that protects the privacy of individual team members. “You don't want to use data to reward or punish an individual,” says Wiegers. For example, if you're using metrics to evaluate code quality in a software development project, avoid using team members' names or singling out specific mistakes made by one person.
By the same token, locally optimized tasks—set up so one member looks like “the good guy”—can be detrimental to the overall structure of the project, says Dave W. Farthing, principal lecturer at the University of Glamorgan, Pontypridd, Wales, U.K. “You need a [critical] chain that is both light and strong. If one link gets filed down, it'll be weak, but if every link gets filed down just a little, the chain will still be strong,” says Farthing.
It's here that so-called “buffer management” comes into play. If a team member regularly finishes a task ahead of schedule, he might start sitting on it until the task deadline to avoid getting overloaded with work. “This guy is used to having a little buffer,” says Farthing. “He's making himself look like a hero because he's finishing every task on time, but because he wastes his own buffer by sitting on the task, none of it is available for use by anyone else. Make the buffer available to someone else, so the project can finish sooner, thereby reducing costs.”
You need a [critical] chain that is both light and strong. If one link gets filed down, it'll be weak, but if every link gets filed down just a little, the chain will still be strong.
DAVE W. FARTHING,
PRINCIPAL LECTURER AT THE UNIVERSIY OF GLAMORGAN,
PONTYPRIDD, WALES, U.K.
“The synonym for value is goodness, and people want to know they're doing a good job,” Rollins says. “If a team member delivers a project 30 days ahead of schedule, that's value. Team members need to know when they are meeting expectations.”
Metrics can play a key role in determining whether a project makes the grade, but cold, hard numbers are the end result of human productivity. PM
John S. Webster is a Providence, R.I., USA-based freelance writer. His articles have appeared in CIO, Computerworld, InternetWeek and Infoworld.
For the latest in Project Management Institute (PMI®) news and project management information, visit PMI's Web site at www.pmi.org.
This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI.
PM NETWORK | NOVEMBER 2002 | www.pmi.org
