The elusive ROI
by Jeannette Cabanis-Brewin
AMEME IS AN IDEA: an idea whose time has come and which, for some mysterious reason, starts popping into people's minds everywhere, all at once. Some have compared it to a virus: a “catching” thought that spreads like wildfire. In 1999, one such contagious strain of thought concerned measuring the return on investment of implementing project management tools and processes.
Everybody's Talking 'Bout …
Return on investment is a measure of profitability computed by dividing the amount of operating income earned by the average investment in operating assets required to earn the income. ROI measures how effectively assets are used to earn income—simple enough in theory. The difficulty arises when the investment is made in an organizational change such as project management implementation. When you change the way people work, it's not as simple to measure the benefits as it is when, for example, installing a new air conditioning system in the company HQ.
Not that this was a new idea: Project managers have been complaining for years that, while they knew project management worked, they didn't have the hard data necessary to prove it by the numbers to their executive leadership. In 1997, C. William Ibbs and Young-Hoon Kwak of U.C.–Berkeley published The Benefits of Project Management: Financial and Organizational Rewards to Corporations [PMI], the results of a study designed to show how organizations benefit financially from implementing the discipline. That study contributed to the debate without resolving it: critics of its findings noted that the pool of surveyed companies was too small and the industry segments they represented too diverse for the results to be really useful. Others were skeptical of the fact that the participating firms self-reported on their level of project management maturity, an area where companies are notoriously overconfident. But Ibbs and Kwak's work added yeast to the mix: within the year, a welter of publications and presentations on ROI began to appear.
Jeannette Cabanis-Brewin (email@example.com) is editor-in-chief of the Center for Business Practices (www.pmsolutions.com/cbp), publishers of the Project Management Best Practices Report and Best Practices E-Advisor. She is the former acting editor-in-chief of PMI's Publishing Division. Direct comments on this article to firstname.lastname@example.org.
Among others, Joan Knutson attacked the topic in her PM Network columns in early 1999 and in her PMI ’99 paper, “Measurement of Project Management ROI: Making Sense to Making Cents,” and Capers Jones reported on a method of measuring ROI on software development projects in the July 1999 PM Network. While on one hand experts tried to develop a numerical formula that would simply and clearly demonstrate the value added by project management, another chorus of voices was rising to protest that such paint-by-numbers formulas were misleading. And—in true meme fashion—the debate was not limited to project management publications.
The February 1999 issue of Training and Development magazine described an Andersen Consulting method of measuring ROI on training projects that “rejects dollar-based ROI and the idea that business benefits from training can be isolated from other factors and proved.” And a September 1999 article in CFO on figuring the ROI on ERP implementation projects—a type of enterprisewide initiative not unlike implementing project management in terms of the organizational change involved—noted that “most people base their justifications on ‘tangible’ benefits alone [and] that's a mistake.” The author, Bronwyn Fryer, warns that CFOs who focus solely on tangible measures like lowered headcount miss out on possible returns. Yet the more intangible benefits—like customer satisfaction—aren't easy to measure. A study of 62 Fortune 500 companies conducted by Benchmarking Partners Inc. revealed that while most companies deploying ERP systems were not able to lower costs as much as they'd expected, they nevertheless experienced better-than-expected results in productivity, cycle time, procurement, and other areas once they'd surmounted the primary barriers of change management, internal staff adequacy, and training.
Project management software industry insiders interviewed in May 1999 (see “The Human Side of Tool Implementation,” PM Network, July 1999) pointed up the same issues at work in major implementations of their tools. The human, cultural, and organizational issues and benefits of implementing a new way of doing business, they said, present a thorny challenge to traditional methods of measuring ROI. When a company invests in project management, the training levels of individuals interact with the maturity level of the processes in the organization and the level of uncertainty in the marketplace to create a volatile workforce working in a very dynamic workplace. This can play havoc with one favored approach to measuring productivity gains: resource utilization tracking. The migration of brain power from one task or project to another when they are freed from rework or otherwise made more effective by new software or a new model of organization throws precise financial measurement of time savings for a loop. Software is racing to catch up and make such calculations feasible but the Catch-22 is that most companies that implement software don't invest enough in training. Frequently project management software functionalities are underutilized by a population of “illiterate users”—a frustrating scenario for the developers of the product. In addition, often companies ask the vendor to implement processes and methodology at the same time as the software. Then, the question becomes: Is the ROI due to the software … or the processes?
At the 1999 Project Leadership Conference in Chicago, a roundtable of IT project managers gathered to compare their experiences with figuring ROI in their organizations. The consensus among those present was that ROI, as presently understood and measured, is not a very good indicator of the value of project management because there were so many intangible corollary benefits that were not easily quantified as dollars and cents. They cited the enormous value of the cultural change, in terms of improved morale and accountability, among other things. They also talked about the “hidden” or at least unforeseen human side costs of implementation in those instances where a company is new to project management and thinks the cost of implementation is the cost of software, period.
Roundtable participant Kevin Bertram, chief of systems development for Oakland County, Mich., called investment in project management “A leap of faith: In IT no one has been able to quantify it. You know how you were operating before and you can see and feel the value. We added to our project office on an assumption of cost avoidance; for example, we surfaced a backlog, but I can't tell you what eliminating it added to the bottom line. Having begun in 1997, I know we now plan better, prioritize better, have more buy-in from customers, and are more business-driven. Plus, project management processes can improve everyone's quality of life … and that's worth a lot.”
Reader Service Number 190
The “Balanced Family” of Measures
The Center for Business Practices is conducting a survey of several hundred organizations that have implemented project management initiatives, seeking to determine the value of those implementations based on a balanced set of measures as outlined below. The measures are a mix of financial, operational, and social measures that show value to stakeholders. Measures in these categories should also be a balanced mix of inputs (money and people put into a process), outputs (results achieved), and processes (performance of the systems that deliver the output). A balanced family of measures would include the following (the exact set would vary with each organization, depending on its strategy):
■ For Shareholders: earnings-per-share growth, cash flow per share, economic profit, total factor productivity, percentage of new-product sales
■ For Customers: customer satisfaction, product quality index, repair incidence, customer loyalty
■ For Employees: employee satisfaction, employee turnover, productivity, diversity in management, training time per year
■ For Community: index of environmental impacts, safety record, community satisfaction, charitable and community contributions.
A subset of measures that focus only on operational issues would include:
■ For Shareholders: productivity, cost reductions, defect level, cycle time, customer retention, employee turnover, percentage of new product sales, process errors
■ For Customers: repair incidence, product quality index, on-time shipping, customer satisfaction, customer retention, customer loyalty, new product inventions, share of wallet, market share
■ For Employees: OSHA recordables, employee satisfaction, absenteeism, employee turnover, empowerment index, grievances, training time per year, competence levels, salary levels, benefit levels, family-support services, percentage of flexible schedules
■ For Community: safety record, salary levels, community satisfaction, legal actions.
Interested in participating in this research project? E-mail Jim Pennypacker, director of the Center for Business Practices at email@example.com for more information.
Reader Service Number 096
ABT's Edward Farrelly agreed with him, calling investment in project management “an employee retention investment”—something especially critical in the present job market. Farrelly also noted that the approximately 1,000 organizations that his company works with, companies that are “relatively serious” about project management, are turning away from purely quantitative measures and using “a value creation theme” instead to justify investment. “By the creation of a project office,” he said, “you free up project managers, unlocking opportunities to create value.”
So it seems that experts, authors, vendors, and practitioners are coming to agree that trying to nail down with mere numbers something as pervasive, revolutionary, and people-powered as project management is akin to using a wrench when you need a hammer. Still, if you are asking a company to spend megabucks for the tools, organizational change, and training that comprise a project management initiative, some sort of quantitative demonstration of the value of project management is needed to help justify the investment. The anecdotal approach is no longer enough. But what can take its place?
If they can come up with ROI formulas for ERP implementation—and CFO's Fryer insists that “many if not most ‘soft’ benefits can correlate to hard numbers, provided project analysts are willing to do some imaginative number-crunching,” then sound data on project management can't be far behind. One project management consultant who has done some of that number crunching is Jim Brosseau, vice-president for client services of the Vancouver-based Software Productivity Centre (www.spc.ca).
“To throw out ROI as a decision-making tool just strikes against what I see as sound PM practice,” he says. “I lean toward using both the quantitative and qualitative measures for the strongest argument possible. Certainly there are a lot of benefits to improving PM practices that are probably too nebulous to quantify, such as improved morale. On the other hand, even things like reduced staff turnover and improved customer satisfaction can be mapped to hard data with a bit of effort and the proper focus. Many other areas, such as reduced rework and improved quality are easy to quantify and can make for a convincing argument. One problem is that one of the major benefits of project management improvement is predictability, and few companies have a good understanding of the cost of their uncertainties.” As an example, Brosseau offered the story of the CEO of one of his client firms, who jumped up suddenly during a requirements management presentation, “like a light bulb had just turned on,” to explain to the staff how misunderstanding requirements on a previous project had cost the firm nearly a quarter million dollars. In working with that same client, Brosseau noted that “while we obtained strong data showing over 100 percent increase in productivity and an ROI on the order of two months, it was the ‘touchy-feely’ stuff that seemed to have the strongest impact: morale was improved, everyone seemed to be working to the same page for a change, the stress of uncertainty was reduced.”
What to Measure? “The challenge for measuring productivity increase,” Brosseau says, “is that nobody has numbers to begin with. It's tough to quantify improvement when there are no baseline numbers to use. As a place to start, a company can get together historical data—not even a year's worth, but just the last project—not anecdotal recollections, but quantitative history using even one or two data points (original estimates, actual effort, number of defects reported). For software development firms, Brosseau has used a formula based on change in the product size over effort expended.
Reader Service Number 186
“Financial accounting … [is] an X-ray of the enterprise's skeleton. But much as the diseases we most commonly die from … do not show up in a skeletal X-ray, a loss of market standing or a failure to innovate do not register in the accountant's figures until the damage is done.”—Peter Drucker.
It's one thing for companies to figure out some workable internal formula, and another, broader challenge for the discipline to come up with ways of figuring ROI that are commonly accepted best practices—which, if they existed, would mean that each company faced with the ROI question wouldn't have to re-invent the wheel. Unfortunately, says Kent Crawford, CEO of project management consulting, training, and research firm PM Solutions Inc., a primary author of an ROI research study being organized by the Center for Business Practices, “There really isn't any substantial significant quantifiable data that evaluates the worth of project management at present.” Like others, Crawford believes that ROI calculations alone are not good indicators of the value of project management—that many other, more intangible (yet quantifiable) benefits will accrue but not show up in ROI calculations. He argues for a broader view of the question similar to the balanced scorecard approach now being used by many Fortune 1000 organizations to measure and manage their organizations.
“Research has shown,” says Crawford, “that creating value for stakeholders is the key to organizational success. Companies that stress shareholders, customers, and employees outperform firms that do not.” He bases his assertion in part on the work of John Kotter and James Heskett [Corporate Culture and Performance, The Free Press/Macmillan, 1992]. According to their research, over an 11-year period, the companies that stress stakeholder satisfaction increased revenues by an average of 682 percent vs. 166 percent for those that don't; they expanded their workforces by 282 percent vs. 36 percent; grew their stock prices by 901 percent vs. 74 percent; and improved their net incomes by 756 percent vs. 1 percent.
“Unlike financial measures,” Crawford says, “most operational measures are leading indicators. When they improve, an improvement in the financials follows, albeit often indirectly, and with a time lag. So to drive success, financial measures must be supplemented with nonfinancial ones.”
The “Balanced Family” Approach. Taking a leaf from the “balanced scorecard” approach [Robert Kaplan and David Norton, The Balanced Scorecard: Translating Strategy Into Action, HBS Press, 1996], Crawford calls his proposed list of measures a “balanced family of measures.” Since deriving advantage through stakeholders drives strategic success, he says, the best way to create measures is to look at which key stakeholder groups most help drive the strategy and devise measures to fit these groups (see sidebar). And the stakeholders most responsible for helping it deliver on its strategy are shareholders, customers, employees, and communities.
LIKE A DOG CHASING its tail, the executive leadership of many companies demand to know research-based details on how project management can benefit them before they spend the money to implement tools or training, while missing the chance to succeed on every round of new projects destined to a rocky road for lack of … tools and training. It's time for project management professionals to band together to create the research basis that businesses need to move to the next level of achievement, instead of leaving project-oriented companies to figure it out piecemeal. ■
PM Network April 2000