Project Management Institute

A measure of success

img

Martha Wellman, Chief Legislative Analyst, Office of Program Policy Analysis and Government Accountability, Florida Legislature, Tallahassee, Fla., USA

{  BY MALCOLM WHEATLEY • PHOTOGRAPHY BY JONATHAN ALLAIN  }

img
img

Tony Blair's government had a big idea: Projects that don't look like they're delivering the goods will be unceremoniously axed. After almost 20 years in opposition, Blair's Labour Party came to power in 1997 filled with reforming zeal. But seven years later, with the public fed up watching high-profile initiatives launch only to see them crash and burn, British government departments increasingly find their projects being scrutinized against a new set of metrics.

img Termed performance-based budgeting (PBB), the system aims to define the benefits that each project is supposed to achieve, describe the measures that show whether those benefits really are being achieved, and stop expenditures on projects that aren't delivering those benefits—or at least, quickly figure out a way to get the project back on track.

For weary British voters, it's not a moment too soon, as government project failures have been all too public. Take the 1999 launch of new systems and procedures at the Passport Agency. Each year, millions of Britons head to the beaches of the Mediterranean for their annual vacations—so when the passport processing time ballooned from 10 days to eight weeks, and the backlog of undelivered passports reached 565,000, thousands of Britons were reduced to spending hours queuing on sidewalks in desperate attempts to secure their passports before the start of their vacations.

EXECUTIVE SUMMARY

img

img Performance-based budgeting can discourage further expenditures on projects that are not going to deliver results in line with expectations.

img Performance-based budgeting can deliver results, as long as executives realize its limitations.

img Executives must clearly define objectives, and project managers must work out how they are to be measured.

img Some people believe performance-based budgeting may be applicable to the private sector

Despite the introduction in 2000 of strict new project management guidelines aimed at preventing such foulups, issues have continued to arise. Just last year, following a tax and welfare reform that targeted low-income families with children, the weekly welfare checks for tens of thousands of families suddenly were interrupted, with claims lost in a miasma of bureaucracy. While newspaper and television reports profiled families and single parents suddenly finding themselves living on a fraction of their former incomes, the government found itself in a messy spat with outsourcing contractor EDS over who was to blame.

img

PBB meant that scarce public resources could be applied to services with high political support, rather than services with high bureaucratic or professional support.

Kerry McGovern,
Consultant for public and financial sectors,
Brisbane, Australia

{devil in the detail}

In a world accustomed to thinking in terms of return on investment, performance-based budgeting ought to be a no-brainer. But the technique is prone to seeing governments bogged down in spats over tiny questions of detail. Ignore these questions at your peril: The measures must be clear, unambiguous, well-defined and—most important—agreed upon by everyone involved.

The measures also must be relevant. For an education-related project, the measures must support the desired objective. If the goal is to improve reading standards, for example, then a measure of the number of books bought would be relevant but not as relevant as reading test scores.

img

Once agreement has been reached, there's still the small matter of collecting the data. “Is there a system in place to efficiently and accurately collect the data?” says Florida, USA, management consultant Kaye Kendrick. “And if not, how much will the system cost—and is obtaining the information worth the cost?”

Unless there is agreement on the collection methodology, the potential for disputes about what exactly constitutes data is enormous. Even supposedly simple things need to be nailed down. Take the collection of data on license issues, for example. “Should the number of licenses issued include re-issues?” Kendrick says. “And what is the definition of a re-issue?”

Even when such hurdles have been passed, knotty problems remain. Who will collect the data? Who will document the methodology? Who will review and validate it? And can the entire population be assessed, or is sampling a practical approach?

Rather than lose sight of project delivery in the detail of milestones and updates, project sponsors—government departments and ministers—can measure their progress and success against big-picture metrics that voters can identify with. Is crime falling? Are education standards rising? Are passport processing times faster? And so on.

Plenty of governments worldwide—at both the state and national level—have experimented with the PBB method because it works. However, it doesn't always work as often, or as reliably, as its proponents hoped.

Ups and Downs

In New Zealand, a country that found itself with a moribund economy and a stifling bureaucracy in the 1980s, the impact of PBB reforms enshrined in the country's Public Finance Act of 1989 have been impressive and well-documented. From being the economic sick man of Australia, the country has become something of a regional powerhouse.

“PBB meant that scarce public resources could be applied to services with high political support, rather than services with high bureaucratic or professional support,” says Kerry McGovern, a Brisbane, Australia-based consultant for public and financial sectors who witnessed the reforms firsthand while living and working in Wellington, New Zealand, at the Office of the Controller and Auditor-General of New Zealand. “PBB became very fashionable because of its success in New Zealand, and other countries have adopted it. But treasuries haven't always realized that the reforms in New Zealand built upon prior reforms, and complemented the strengths of the people in government at the time.” The end result often is PBB in name, but not in delivery.

Take Florida's school system, for example, where PBB is in force. It has performed both well and badly, says Martha Well-man, chief legislative analyst with the Florida Legislature's Office of Program Policy Analysis and Government Accountability. “It's performed well because test scores are rising, indicating improvement in basic skills such as reading, writing and math,” she says. “But some people argue that the improvement has come at a cost—too narrowly focusing on just those subjects and taking away other valuable education opportunities.”

img The root problem is familiar to those well-versed in PBB, and one that managers everywhere—private and public sector—can recognize. Put bluntly, you get what you measure. And if what you're measuring isn't well-thought-out or well-measured, beware—the results may not be helpful at all.

Good Measure

PBB works well in areas where outcomes can be reasonably measured, such as health care, social services, corrections and so on, Wellman says. However, the practice shows far lesser success in areas where outcomes either cannot be reasonably measured or where scrutinizing them could produce undesirable side effects.

img

{top PBB tips}

imgDecide what you want your project to achieve

imgDecide what measurements will show

imgCollect the data

imgValidate the data

imgDetermine if the project has done what was expected

imgConsider reallocating resources elsewhere if it hasn't or won't.

img

“Examples are the justice system—how do you measure justice?” Wellman says. “And in transportation, there isn't a measure for mobility. Focusing on measures for highways such as congestion indicators can lead to building more highways when improved multimodal transport systems might be better. PBB is also more difficult for activities with uncertain long-term outcomes, such as basic research.”

img

I might have a portfolio of a hundred projects but only do the 30 with the highest return on investment.

Art Drake,
Director, Business Assurance Services,
Upstream Solutions,
Minneapolis, Minn., USA

img Even where conceptually sound measures on which to base PBB exist, the process of measurement itself can be flawed, says Kaye Kendrick, a Tallahassee, Fla., USA-based management consultant with extensive PBB experience. PBB has been characterized by wrangles over what to measure, why and how. “Data collection methodologies haven't been put in place, they haven't validated the data, and there has been no overall structure of measures,” she says. “The result? Everybody was constantly changing the measures that got put in the budget.”

Even more damning, adds McGovern, the process of measurement itself consumes resources. “PBB generally sees funds pushed toward computerized data collection, collation and storage systems that may not be efficient in the long run,” she says. In instances where resources are inadequate or under pressure, she adds, the additional burden of data collection and reporting can divert funds into administrative tasks and away from necessary public services such as water, sewage, health and law enforcement. In other words, weighing the pig doesn't make it get any fatter.

img Despite these shortcomings, PBB has much to offer—even in the private sector—says Art Drake, director of business assurance services at Upstream Solutions, Minneapolis, Minn., USA. To Drake, PBB imposes a discipline on an organization's portfolio management, enforcing both a rational project approval process and a periodic review of whether anticipated outcomes are likely to be met.

“I might have a portfolio of a hundred projects but only do the 30 with the highest return on investment,” he says. “Then, when I review them, I might find that projects ‘A’ and ‘C’ aren't going to meet their ROI objectives, or miss the market opportunity through being too late. So the program management office can decide to either start another project or reallocate the resources elsewhere.”

Even so, Drake's enthusiasm is tempered by a familiar qualification. “You have to have the right measures in place to make sure that the numbers you're looking at equate to value—because you're going to be making important decisions based on those numbers,” he says.

Uncertain Future

Back in Britain, it's far from clear that Drake's warning is being heeded.

“Too many of the measures that are being proposed rely on self-assessment, and you have to go into a lot detail within the measures to determine how well projects are actually being managed,” says Alan Harpham, chairman of Britain's The APM Group, a project management accreditation, examination and certification body with extensive public sector involvement.

Harpham's concern: Charged with measuring the effectiveness of their projects themselves and using their own metrics, British civil servants may be tempted to fudge the figures. PM

Malcolm Wheatley is a U.K.-based journalist and regular contributor to magazines such as CIO and MSI.

This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI.

PM NETWORK | JUNE 2004 | WWW.PMI.ORG

Advertisement

Advertisement

Related Content

Advertisement