For a true read on project success, organizations must track both qualitative and quantitative benefits.
Project managers typically consider a project successful if it meets quantitative goals—on budget, on time and within scope—set forth during the planning phase. Qualitative benefits, such as stakeholder satisfaction or product aesthetics, often are passed off as secondary. That's a big mistake, according to Carlyle Maranhao, PMP, Hewlett-Packard Co., Chester Springs, Pa., USA, and Christine Green, PMP, EDS Denmark, Copenhagen, Denmark. The two discuss why qualitative benefits are just as important as the quantitative ones.
Should organizations build metrics into their project management process? Why do some project offices fail to measure project results?
Ms. Green: Organizational metrics programs take a bit of time to implement. It's not done overnight, so that's one of the issues. The other is a lack of management sponsorship. They don't see the benefit.
Mr. Maranhao: Any organization that has gone through the effort of establishing a project management office (PMO) typically has defined metrics for all the projects it manages. If we were talking years ago when PMOs were kind of new—when everybody wasn't sure how they should measure—the answer might be different. But PMOs have matured in terms of measuring projects results.
Ms. Green: If they don't, they don't succeed.
Mr. Maranhao: They'll be fired!
How can companies measure quantitative and qualitative benefits?
Mr. Maranhao: Both benefits are empirically measurable, it's just a matter of which techniques you choose to use.
Points of View
CARLYLE MARANHAO, PMP, is a client principal with Hewlett-Packard Co., Chester Springs, Pa., USA. His 17 years of experience in the IT industry includes consulting for several Fortune 500 companies, program management and sales.
CHRISTINE GREEN, PMP, is a metrics and estimating specialist at EDS Denmark, Copenhagen, Denmark. She is a certified function point specialist and software measurement specialist and is on the IT Performance Committee of the International Function Point Users Group.
Both quantitative and qualitative benefits have strengths and weaknesses. The power is in using them both at the same time.
–Christine Green, EDS Denmark, Copenhagen, Denmark
There are two sides of the equation. One measures direct return on investment (ROI)—typically there's a lot of focus on what the project is going to save the company when it's delivered. But the other side of the equation that's often not directly quantified deals with the optional benefits that may not be immediately apparent.
Real options analysis is a technique that can be used to measure the qualitative benefits of a project in its beginning stages. It's based on Black-Scholes options techniques, which are used everyday in the stock market to quantify the value of stock options. This technique is an empirical method that gives you a way of quantifying the potential future value of a project when that value is not immediately apparent.
Many projects that might fail the test from a straight ROI perspective are actually good investments because of the “optional value” side of the equation you need to measure.
Ms. Green: Traffic light reports also are a very effective way because that's something the executives intuitively understand—if it's red, they need to have somebody look at it. I don't necessarily have to do it as an executive, but I know that if it's red, somebody needs to.
Mr. Maranhao: One of the interesting things I've found about traffic light reports is there's a natural reluctance for project managers to show anything as red. So if you see a green project it will be green, green, green and then all of a sudden, it will completely fall into red because green is good, red is bad. You have to get more empirical about how you measure what constitutes yellow and red and green.
A strong business case is made up of both the qualitative and quantitative side.
–Carlyle Maranhao, PMP, Hewlett-Packard Co., Chester Springs, Pa., USA
What are the strengths and weaknesses of tracking qualitative benefits versus quantitative ones?
Ms. Green: Both qualitative and quantitative have strengths and weaknesses. The power is in using of both of them at the same time. You can't say “use one over the other”—they both provide important feedback. If we were building a bridge, for example, the qualitative benefits relate to whether the bridge works, but the quantitative might measure how much concrete was used or how many people were working on it. All of this information is important.
Mr. Maranhao: A strong business case is made up of both the qualitative side and the quantitative side. To use the bridge example, if it was built on time, on budget, and met specs but wasn't beautiful to look at, it may not have enhanced the city it was in. It's important to consider the quantitative sides of the bridge—Can it accommodate the necessary number of people? Does it connect the two land masses it's supposed to connect?—as well as the qualitative side—Does it enhance the landscape?
Should qualitative benefits count toward ROI?
Mr. Maranhao: Absolutely. There are two areas of benefit you need to consider when you're looking at any particular project. There are immediate returns as well as the areas of optional benefit. A lot of times those areas of optional benefit can represent a significant area of value to the business. Even more importantly, the process by which you quantify those optional benefits demands the project team seeks out the key stakeholders and gets input from those stakeholders. By doing that, you're making sure you're solidly linking the project to the specific business goals and the areas of focus.
Ms. Green: The one big thing I see when we mention ROI is people are thinking about money. They're thinking about what it gives the company if we implement the project from an earning perspective. They're not thinking about what it will bring them from a qualitative perspective, which could be a performance improvement due to process improvement, a quicker learning curve for their staff members or faster turnaround when information is exchanged.
Are quantitative benefits given too much importance over qualitative ones?
Mr. Maranhao: That's an interesting question. I would say many times yes. Often the value of what the project is delivering is understood better by the business itself than by the project team.
However, if you take a look at many project monthly status reports, you'll often see that the project team focuses on how well the project is delivering. Is it on time? Is it on budget? But often there isn't any inclusion of how well the project is actually satisfying the key stakeholder needs.
Ms. Green: Stakeholder satisfaction is a task that we need to perform, but it also is a task that unfortunately has some difficulties in being consistently used by project teams. It might be used on an organizational level but a lot of projects reject or forget to ask their clients or their users if they're satisfied and the project met their needs. PM
PM NETWORK | SEPTEMBER 2006 | WWW.PMI.ORG