The same old metrics won't cut it anymore. A new triumvirate of project metrics has risen to the forefront to take its place beside ROI: customer satisfaction, human resources (HR) and risk management.
The down economy has certainly affected how metrics are viewed, says Kjell Rodenstedt, PMP, senior partner, OPG Management, a Stockholm, Sweden-based organizational project governance consulting firm. “But metrics are also a way for an organization to improve its maturity,” he says.
YOU SAY YOU WANT AN EVOLUTION?
Even in an economic climate where every dollar, euro and yuan counts, there's an inherent problem with narrowly focusing on ROI as a metric, says Bruce Webster, founder and principal at Webster & Associates LLC, an IT risk management firm in Englewood, Colorado, USA. It's easy to assume a project is ineffective or failed when only its direct contribution to the bottom line is considered.
Instead, organizations should take a broader view of financial metrics. “Will the project reduce costs? Are we doing this project because if we don't, we'll fall behind the competition and become less effective in the marketplace?” he asks.
Financial success measurements aren't the only ones to track. In fact, customer satisfaction-driven metrics have become the number-one priority for many organizations, says Bijan Nikravan, PhD, PMP, director of project management excellence for IT giant Microsoft in Chicago, Illinois, USA. “These metrics may have previously not been given as much attention,” he says. Effective customer satisfaction metrics include measuring return business, the number of customer objectives met, controlled disruption to normal business while projects are underway and adhering to quality standards that were originally agreed upon.
Work health index and organization health indexdriven metrics are also becoming increasingly important. These KPIs measure an organization's health in terms of individual and team morale, job satisfaction and the like.
“With the downturn of the economy, organizations went through a period of losing high-potential project staff,” Dr. Nikravan says. As organizations recover and attempt to get back to “business as usual,” the need to measure employee retention and satisfaction increases.
Take a more proactive approach than simply conducting an employee satisfaction survey once a year.
“Companies such as Microsoft set employee satisfaction metrics and retention targets at the beginning of each year, and they make ongoing adjustments based on business and market conditions,” Dr. Nikravan says. “If you have human resource issues, you can't wait until the annual survey results come out.”
Equally important today are risk management-related metrics, says Cindy Margules, PMP, project management office director at Convio, a software as a service company serving not-for-profit organizations, headquartered in Austin, Texas, USA.
Organizations must look beyond merely the number of risks. “Just because you have identified 300 risks doesn't mean the project is riskier than a project with only one risk,” she says.
Instead, organizations should track:
- How often a risk assessment is completed
- The number of risks that are rated low initially but become issues later
- How often risk assessments are reviewed
- The percentage of actual risks that were identified in advance
DO YOUR METRICS PASS THE TEST?
The need to put hard numbers to project progress—and ultimately portfolio success—is twofold: Without solid metrics that relate to key performance indicators (KPIs), project managers and team members have a difficult time justifying their work, while executives remain in the dark.
It's a classic syndrome, Mr. Webster says. A lack of tangible metrics means bad news doesn't get reported upward. “No one wants to go to the boss and tell him or her that things are really messed up and the team is going to miss the deadline,” he says. “At the top rank, everyone thinks everything is fine. But down the hierarchy, the project team knows that's not the case.”
Metrics help those above the daily trenches definitively know where a project stands—good or bad. But not just any measurement will do. Mr. Webster says metrics should pass a three-part litmus test:
▪ Metrics must be informative and predictive. They need to provide relevant, useful information about what has been achieved or when a milestone will be met. That sounds like common sense, but plenty of project professionals don't follow this rule, Mr. Webster says.
“One of the most popular IT project metrics—and one of the worst—is source lines of code,” he explains. “You end up with a large, bloated code instead of something that achieves what you really need it to. Project teams will want to achieve the desired metrics whether or not the metrics truly measure project success.”
▪ Metrics must be objective. They must be calculated consistently and repeatedly, independent of who's doing the measuring. “Another popular project metric is the ‘walking around’ metric,” Mr. Webster says. “But if you walk around to team members and ask them how far along they are, they're just going to pull a number from their hat—and most likely a number that makes them look good or avoid blame.”
▪ Metrics must be automated. If a metric requires a lot of manual work to be measured, the project manager or team member responsible for it will eventually abandon it. Organizations should consider how much time is spent tracking the metric—it might not be worth the effort.
THE TROUBLE WTH METRICS
Project metrics can be as vital as they can be imperfect. To get the most out of hard numbers, organizations must remember:
1 Metrics are often backward-looking. “If organizational leaders have a micro focus on KPIs, they could miss opportunities or move the organization into failure because they aren't looking ahead,” says Cindy Margules, PMP, Convio, Austin, Texas, USA. Keep in mind that metrics aren't the big picture—they're just a piece of it.
2 You can't measure everything. Certain project characteristics are intangible and therefore can't be associated with a metric. “For example, you can't measure feelings. You might have team members tell you their feelings, but you can't measure that as a KPI,” says Imad Alsadeq, PMI-RMP, PMP, Thiqah, Riyadh, Saudi Arabia.
3 Not all metrics are worth the effort. “You shouldn't spend more than what you get from the measurement,” Mr. Alsadeq says. If a metric takes a significant amount of time and effort to track, it's probably not the right metric in the first place.
4 Metrics shouldn't be used to evaluate project managers and team members. If someone's job depends on achieving a certain cost savings within a given project, he or she will try to accomplish that goal—even if doing so harms other aspects of the organization, says Bruce Webster, Webster & Associates LLC, Englewood, Colorado, USA.
5 Negative behaviors can manifest from certain metrics. Ms. Margules worked with one project team that was given a KPI of not “overserving” clients, measured by hours dedicated to each. “But in order to not overserve, people were not submitting their time spent with clients,” she says. “The organization ended up unable to accurately measure work.” It realized that while the KPI didn't need to change, the metric used to measure it did.
Don't forget about earned value management (EVM) metrics says Bijan Nikravan, PhD, PMP, Microsoft, Chicago, Illinois, USA. Measure the planned value against earned value on a regular and ongoing basis, which in most cases is weekly.
“It is also important to then communicate these metrics with the project team and stakeholders.”
KEEPING UP WITH KPIs
“In a tough economy, knowing if a project is on track can make the difference between profit and loss,” Mr. Webster says. “Gone are the days of launching projects within a portfolio with unclear objectives, poor scheduling and cost estimates, and no way to track progress. For those at the higher level, they can't afford that anymore.”
An organization must create the right portfolio KPIs, and then determine the most effective project metrics for measuring those indicators.
“You can't manage what you can't control, and you can't control what you can't measure,” says Imad Alsadeq, PMI-RMP, PMP, consultation director at Thiqah, a management and engineering consultancy in Riyadh, Saudi Arabia.
The secret to choosing the correct KPIs is quite simple: Ensure that they're built on the organization's strategic objectives. “You have to think of the justification for having the projects or project portfolio,” Mr. Alsadeq says. “That justification will tell you what you should be measuring.”
Hospital executives, for example, might establish a KPI to increase the number of operations. “More operations might bring in lots of revenue, but over time, the hospital might develop a bad reputation from expanding too quickly,” he says. “Instead, the hospital needs to revisit its vision or mission: Is it looking for money? Or is its goal to heal patients and help society?” A better KPI should speak to the hospital's strategic objective to increase the percentage of successful operations.
Creating the right KPIs could make or break an organization. “The wrong KPIs will drive the organization to an undesirable goal. Efforts will be wasted, and project teams will perform inefficiently,” Ms. Margules says.
From the beginning, make sure the right people are involved in the decision-making process. Get input from those who will be tasked with achieving the benchmarks, including project managers and team members, Mr. Alsadeq suggests.
Not involving those in the trenches creates unnecessary risk—as Mr. Webster witnessed on a project to launch a new customer-facing website.
“The company signed a contract with an outside vendor to provide an inexpensive, out-of-the-box system without talking to its internal IT department,” he says. In the end, that poor decision meant the IT department had to invest significant time and effort to address quality issues with the system.
On the surface, the project met the organizational objective for a low-cost system. Had the IT department been involved early on, though, a more relevant KPI would likely have been determined.
Even the “right” metrics can change over time. Some that were appropriate a year ago may no longer be today. And who better to recognize when a metric is no longer working than those responsible for tracking it?
“Organizations should not only collect metrics, but after a few months, they should also collect feedback on metrics,” Ms. Margules says.
This can be done via an anonymous survey or one-on-one meetings, asking project managers and team members such questions as, “How well is this metric working for you?” and “What behaviors are you seeing with the project team as a result of this metric?”
“It's with this kind of feedback that the right adjustments can be made,” Ms. Margules says.
Above all, choosing the right KPIs and metrics starts in the executive suite. “If a company doesn't have an overarching goal that everyone can say, ‘This is what we are working toward,’ then people won't be moving as one unit,” she adds. “Most organizations don't realize that if they can be crystal-clear on one goal, they can drive exceptional performance.” PM
EXPLORE MORE ON METRICS IN A CLOSER LOOK
PM NETWORK MAY 2011 WWW.PMI.ORG