R&D performance measurement and benchmarking

Many companies have made significant progress in their product development capabilities over the last decade. However, many are still finding it difficult to assess their overall product development performance and the return on investment of their R&D expenditures.

This paper highlights the results of a benchmarking study that PRTM conducted via its subsidiary The Performance Measurement Group (PMG) with over 40 companies across different industry segments. It identifies the main reasons why in-place measurement programs are not working and why companies are not reaping the full benefits of product development performance metrics. Finally, it introduces PRTM's structured-implementation approach to building a successful metrics program.

HOW TO ASSESS PRODUCT DEVELOPMENT CAPABILITY: THE PRTM MATURITY MODEL

Over the past decade, companies have made significant progress in their product development performance. By using a best-practice approach such as PRTM's PACE^® (Product And Cycle-time Excellence) methodology, many companies have climbed the ladder of product development maturity to achieve better performance overall.

Our extensive research and client work consistently show that companies evolve through different stages of product development maturity. Each stage represents a step-level change in performance, not continuous improvement. Moreover, companies must make fundamental structural changes to progress from one stage to the next.

PRTM's Stages model characterizes the evolution of a company's product development capability:

PRTM's Stages of Product Development Capability

Stage 0, informal management, is characterized by an absence of formal product development practices of any kind. A company at this stage of maturity has no standard, repeatable processes for executing a product development project. Although seemingly low on the capability scale, this stage characterizes some very successful companies—most notably start-ups.

Start-ups generally have very few people, so communication is less complex. They usually have a singular and narrow focus, so neither process formality nor standard operating procedures are needed for effective product development.

But as a successful start-up grows, a strong foundation of functional capabilities is critical. Strength in such key areas as engineering, manufacturing, testing, and marketing will position the company to move forward from Stage 0 to Stage1, where functional strength is the main focus.

Stage 1 companies tend to do fairly well until they reach revenues of $50 to $70 million. At this point, they begin to hit a wall of increasing complexity, as multiple functions attempt to manage multiple projects at the same time. Determining project priorities becomes a growing challenge, and one function's priority may be another's back-burner item. Scheduling tasks and resources also becomes increasingly difficult as projects grow in number and complexity.

The solution for these companies is to develop a cross-functional capability, which characterizes Stage 2, project excellence. In this stage, companies take a project view of the world, aligning all functions for effective project execution throughout the product development process. Cross-functional and concurrent product development are the cornerstones of Stage 2.

Once project excellence is in place, the next step is to institute the same processes across all projects to reach Stage 3, or portfolio excellence. Portfolio excellence requires a common framework for product development across all projects. In Stage 3, companies are able to achieve platform leverage and portfolio balance, which are key to product strategy. Stage 3 processes allow management to select which projects to fund and which projects to cancel in order to achieve strategic alignment and to optimize the return on portfolio investment.

With portfolio excellence in place, the next step is Stage 4, co-development excellence. In this stage, companies are able to develop products with selected external partners using collaborative development processes. Besides boosting innovation and productivity, co-development allows companies to focus their resources on the areas that are strategically critical to the business.

BENCHMARKING PRODUCT DEVELOPMENT PERFORMANCE—WHY CURRENT METRICS PROGRAMS DON'T WORK

Using this Stages model, PRTM and PMG recently conducted a detailed benchmarking survey on the product development, pipeline, and portfolio management practices of about 40 companies across multiple industry segments. The study examined the relationship between overall product development capability, the maturity of portfolio management practices, and the level of product development performance as measured by a set of key performance indicators or metrics. Based on our analysis of the results, the most advanced companies in terms of pipeline management and overall product development capabilities (Stages 3 and 4), achieve:

• Two times the industry average for return on investment
• A 90% project success rate
• 25% profit margins as compared to 12-18% for lesser performers
• Faster growth overall

Benchmarking allows companies to objectively assess and quantify their product development performance, and compare that performance against others in the same industry. However, our study revealed that many companies struggle with which specific areas to measure, and how to get the most out of those metrics. In fact, our study found that even companies that have been measuring their performance for some time are failing to reap the expected benefits. Four root causes underlie the problem:

• Performance metrics are not used to their full potential.
• Metrics used are not the right ones—they're not consolidated or balanced. Companies focus too much on project and historical metrics, and not enough on portfolio metrics.
• Tracking mechanisms are inadequate.
• Bottom-line implications of performance improvement programs are not considered.

Performance metrics are not used to their full potential

We asked survey participants how they use performance metrics. As shown in Figure 1, over 70% of companies use metrics to review projects. But companies are not using metrics in a broader manner. Only 55% of the companies polled use metrics for planning and goal setting, 41% use them for benchmarking, and only 38% use them to link their strategy to individual goals. These numbers show that although most companies are tracking metrics, few companies are leveraging the full potential of metrics.

Figure 1: Reasons to Track Metrics

Metrics used are not the right ones

We asked respondents what types of metrics they were using [Table 1]. As shown in Figure 2, companies are collecting a wide variety of metrics, but only 35% of the respondents are systematically tracking consolidated/balanced scorecards. Over half (56%) of the companies are systematically tracking project metrics, but only 21% track technology metrics and only 26% track portfolio metrics. The implication is clear—companies are not getting a full, balanced picture of their product development performance. For example, companies that focus mainly on project-level metrics typically make sub-optimal product decisions, since portfolio and technology-level metrics are absent. A key reason for this imbalance is that most companies rely on a single function such as marketing or product management to measure product success (Stage 1) and have not achieved full Stage 2 Project Excellence capabilities, where management focuses on overall measures such as time-to-market and schedule slippage. Leading companies that operate at Stage 3 Portfolio Excellence have technology and portfolio metrics, such as overall project performance reported consistently across projects, which lets management focus on aggregate portfolio performance.

Table 1: Example Metrics

Figure 2: Types of Metrics

Another issue we've identified is that companies lack predictive measures—those leading indicators of product performance. Historical metrics such as actual time to market are useful for measuring performance and analyzing trends, but predictive measures such as project complexity and project risk can help companies anticipate product development performance problems and take corrective actions. We asked survey participants to list the “pains” they associate with metrics tracking, and 80% named “lack of predictive measures” as an issue [Figure 3].

The data also showed that big or moderate issues for nearly 70% of respondents were “standardizing the data” and finding it “too time consuming” to track metrics (Figure 3). Companies without a standard set of appropriate metrics in place often struggle in these two areas. Without standardized—and therefore comparable—metrics, companies cannot interpret results and determine the appropriate actions to take.

Figure 3: Issues Associated with Metrics Tracking

Tracking mechanisms are inadequate

In the last few years we've observed that companies are using enterprise-wide systems to monitor product development performance. Development chain management (DCM) and product lifecycle management (PLM) systems automate metrics data collection and reporting. We asked survey participants how they collect and report metrics data. Only 24% of respondents are using enterprise-wide systems to track metrics [Figure 4]. Companies with enterprise-wide systems experience fewer problems such as difficulty in standardizing the data [Figure 5] than companies that haven't adopted the systems. Clearly, companies with support systems are benefiting from them, but the adoption rate of these systems is low overall.

Figure 4: Enterprise-Wide System Adoption

Figure 5: Issues Comparison

Bottom-line implication of performance improvement programs are not considered

The survey showed that companies are making progress in using metrics programs as part of ongoing product development management. But although 68% of respondents [Figure 6] have incorporated metrics programs, only 47% [Figure 7] are measuring the financial impact of improvement programs. Without measuring the outcome of improvement efforts, companies have no way of knowing which initiatives have the highest impact.

Figure 6: Purpose of Metrics Programs

Figure 7: Companies that Track the Bottom-Line Impact of Improvement Opportunities

PRTM APPROACH TO BUILDING EFFICIENT METRICS PROGRAMS

As noted earlier, companies with the most successful metrics programs take them seriously, and often implement them as part of a broader quality initiative. To make metrics work, a company should use a product development performance scorecard that contains a balanced mix of metrics in three key product development areas: 1) Project execution; 2) Portfolio and pipeline management; and 3) Product strategy and technology management.

Project execution

Based on our experience, we recommend that the following five key metrics be used to measure project execution: Time to market, schedule slippage, cost variance, time to profitability, and project performance to goal. The table below presents the definition and calculation of these metrics:

Table 2

Portfolio & Pipeline Management

Greater efficiency in portfolio and pipeline management gives a competitive edge to Stage 3 companies. By regularly comparing the mix of products in their portfolio to that of others in their industry, Stage 3 companies can ensure that they have enough platforms and major projects to drive growth. They can also compare their R&D productivity against their total R&D spending.

Key portfolio and pipeline management metrics include new product revenue, R&D effectiveness index (RDEI), and pipeline throughput. Definitions of these metrics are summarized in the table below:

Table 3

Product Strategy & Technology Management

Stage 3 and Stage 4 companies regularly review their product strategy to make sure it is aligned with their overall business strategy. They see technology management as a cross-functional process, and technology development projects as deserving the best practices of project execution excellence. Key metrics include strategic balance of released projects, technology readiness, technology leverage, and market renewal.

Table 4

Implementing a Successful Metrics Program

Finally, to resolve the four root causes of metric program failure, and to build a metrics program that works, PRTM recommends the following 10-step approach:

Table 5: 10-Step Performance Measurement Approach