Measuring What Matters

When projects come into existence, the planning process undoubtedly starts with what needs to be done, when it will be done and how much it will cost. These dimensions frequently form the foundation for tracking progress and reflecting on project success as well. The iron triangle (see Figure 2) is easy to understand, quantifiable, and trackable. However, what it doesn’t do is provide a robust evidence base for the wider impact of projects. The conversation about project outcomes, which happens during the planning process, needs to start driving measurement at project initiation—not afterward. The ability to link performance metrics to outcomes from start to finish is critical in positioning PMOs as valued contributors to strategy delivery.

The Top 10 Percent organizations are more likely to be aligning key performance indicators (KPIs) to wider organizational strategy, communicating the impacts of projects, and making the most effort to improve the line of sight for the C-suite (and engage them in the PMO) during the COVID-19 pandemic (see Figure 3). We believe that this is driving a more outcome-focused approach to measurement. Our research confirms these organizations have a more innovative approach to measurement using more metrics and technology. In order to qualify in the Top 10 Percent, we know they are also more likely than not, to be communicating the impacts of projects and aligning KPIs to wider organizational strategy. That said, there is still more to be done even by the Top 10 Percent to improve the approach to measurement and the use of technology to facilitate measurement.

Our research identifies these four key approaches to improving how the impacts of projects and programs are measured:

1. Increasing the number of measures and frequency of measurement.
2. Increasing the effectiveness of measures.
3. Involving the right stakeholders.
4. Increasing the use of technology.

In our global survey, only 12 project professionals out of 4,069 told us that their organization had no formal metrics in place to monitor the success of projects. The rest, on average, had 7.1 metrics in place for every project—but of these, typically four were linked to the iron triangle (see Figure 4). Therefore, only three additional measures were being used to tell a compelling story of project success. To amplify the impact of projects, and by extension the PMO, additional measures such as societal or environmental impact or return on investment must be the headline act. Being on time, in scope and on budget are the basics; they do not evidence the real change in outcomes that projects can provide.

Overreliance on the iron triangle is also affecting the image of project managers. When participants in our global survey were asked to describe how those outside of the profession would describe project managers, scheduler was the second most frequent word used, selected by 54%. However, among those in the Top 10 Percent, scheduler was just one of many descriptors used to the same degree to describe project managers. In addition, many more participants in the Top 10 Percent used words like changemakers, essential, and realizing visions to describe project managers, compared to organizations overall. This suggests that increasing the scope of measures can also influence the perceived value of project management capabilities and help combat the talent crisis.

The frequency of measurement also poses a challenge. Our findings showed that only 41% of PMOs were consistently measuring and reviewing their performance, and that just over half were regularly communicating project milestones and impacts to the C-suite. Both of these elements are key to improving PMO maturity in our index—using measurements to assess the performance of projects and PMOs has to be done on a regular basis.

There are two aspects to consider when it comes to the effectiveness of measures: firstly, selecting the right measures and secondly, how to implement them.

Selecting the right measures: A measure that’s useful for one project may not necessarily be the best measure to gauge success in the future. Scope, schedule and cost will always be constants, but the outcome measures need to evolve. Measures have to be actively reviewed by the right people—there needs to be purposeful, collaborative reflection on what the measures are saying to tell the story of project impact in the best way and inform future projects. This reflection has to become commonplace—part of daily team meetings, part of project reviews and part of monitoring PMO performance.

How to implement measures: Customer satisfaction is a score that can be calculated in a number of ways, but the meaning is universally understood. It can also be implemented and quantified through a carefully designed follow-up survey, as Sam Farid, Global IT PMO leader for Fonterra explains (see case study). Conversely, cultural shifts (the willingness of employees to adopt a new process) or operational efficiencies present a bigger challenge; the metric needs to be defined and then presented in a way that’s easy to understand. Like customer satisfaction, the threshold for success needs to be defined, but what that threshold should be is likely to be less obvious. For example, tracking adoption and monitoring behavioral change needs further context through employee and customer consultation, but PMOs can also add further value by linking this to wider project data through project and program management (PPM) software.

In order to select the right measures, the right stakeholders need to be involved. Our research reveals only 63% of organizations with a PMO actually engage the PMO in the development of success metrics, and only 39% engage the C-suite. Measures being developed without collaboration between the PMO and senior leadership risk the development of metrics that fail to align to the organization’s strategic vision. The Top 10 Percent organizations further refine measurement by involving a greater variety of stakeholders, including customers, consultants, and external stakeholders, to ensure metrics focus on the outcomes that really matter.

One third of the participants in our global survey stated that their organizations were using technology to effectively measure the impact of projects and programs. Strategy execution management technology is helping these organizations capture more metrics beyond the iron triangle—and on average, 8.3 metrics, compared to just 6.5 metrics among those not using technology to assist measurement.

While working remotely during the pandemic, 79% of project managers in organizations using technology to aid measurement said it was easy to effectively measure project/program impacts. This compares to 63% in organizations not using technology to assist in measurement. Over one third of participants in PMOs that are using technology for measurement stated that their PMOs are very successful, versus 21% within PMOs that aren’t using technology for measurement.

It’s not just PMOs that benefit. Where technology is being used to measure the impact of projects and programs, the organization as a whole saw better performance in 2020 versus 2019 in terms of revenue, environment, social and governance (ESG) indicators, customer satisfaction metrics, and customer acquisition levels.

However, even where organizations are seeing a benefit to the use of technology for measurement, they still face similar barriers to investment in technology. Overall, 54% told us budget constraints are a key blocker to investments in technology; this constraint could come from a range of sources—the purchase price, maintenance, or training. Cost, as a barrier, is true regardless of how much organizations are already using technology to facilitate measurement.

CASE STUDY: Fonterra Co-operative Group

Sam Farid headed up the revival of Fonterra’s centralized IT PMO four years ago. As part of this initiative, the PMO invested in implementing a project portfolio management (PPM) software, with a key focus, at the time of interview, on increasing the visibility of project and portfolio information utilizing the software.

The PPM software brings a range of benefits, including the management of:

• Ideation
• Tracking and reporting
• Planning and collaboration
• Finances

Bringing this functionality together creates a single source of truth, and brings together real-time data on demand to enable better investment decision making. “It’s about trying to get people to move away from their spreadsheets,” says Sam. “Spreadsheets create a nightmare for a single source of truth, so we’re trying to get people to move away from their own measure to a centralized system.”

Understanding if the tools, processes, and frameworks provided by the IT PMO are helping to create value is key to measuring the success of the PMO. Sam and his team measure success through customer satisfaction: “If they are not happy, they are not satisfied—then it’s no good!”  Customer satisfaction is measured and responded to in a number of ways:

• Annual survey: Sent to project managers and key stakeholders in the wider PMO community to understand the level of satisfaction with the PMO. They evaluate different areas of the PMO, for example onboarding and communication.
• Review of annual PMO roadmap: An annual PMI work plan (roadmap) is established with input from the annual survey and other key sources, such as strategy and operating model direction. The roadmap outlines the set of prioritized actions and improvements to be made over the next period, which can be up to two years to uplift customer satisfaction. This also enables the PMO to define how success will be measured in the coming period, for example, KPIs and Objectives and Key Results (OKRs). 
• Post-implementation review (PIR): After every project, the project team and stakeholders complete a qualitative and quantitative review, using a defined template established by the PMO. Output from the PIRs will feed into the PMO roadmap and improvement plan.