R&D performance measurement and benchmarking

55 avenue Hoche
75008 Paris
France
[email protected]

Many companies have made significant progress in their product development capabilities over the last decade. However, many are still finding it difficult to assess their overall product development performance and the return on investment of their R&D expenditures.

This paper highlights the results of a benchmarking study that PRTM conducted via its subsidiary The Performance Measurement Group (PMG) with over 40 companies across different industry segments. It identifies the main reasons why in-place measurement programs are not working and why companies are not reaping the full benefits of product development performance metrics. Finally, it introduces PRTM's structured-implementation approach to building a successful metrics program.

HOW TO ASSESS PRODUCT DEVELOPMENT CAPABILITY: THE PRTM MATURITY MODEL

Over the past decade, companies have made significant progress in their product development performance. By using a best-practice approach such as PRTM's PACE® (Product And Cycle-time Excellence) methodology, many companies have climbed the ladder of product development maturity to achieve better performance overall.

Our extensive research and client work consistently show that companies evolve through different stages of product development maturity. Each stage represents a step-level change in performance, not continuous improvement. Moreover, companies must make fundamental structural changes to progress from one stage to the next.

PRTM's Stages model characterizes the evolution of a company's product development capability:

PRTM's Stages of Product Development Capability

img

Stage 0, informal management, is characterized by an absence of formal product development practices of any kind. A company at this stage of maturity has no standard, repeatable processes for executing a product development project. Although seemingly low on the capability scale, this stage characterizes some very successful companies—most notably start-ups.

Start-ups generally have very few people, so communication is less complex. They usually have a singular and narrow focus, so neither process formality nor standard operating procedures are needed for effective product development.

But as a successful start-up grows, a strong foundation of functional capabilities is critical. Strength in such key areas as engineering, manufacturing, testing, and marketing will position the company to move forward from Stage 0 to Stage1, where functional strength is the main focus.

Stage 1 companies tend to do fairly well until they reach revenues of $50 to $70 million. At this point, they begin to hit a wall of increasing complexity, as multiple functions attempt to manage multiple projects at the same time. Determining project priorities becomes a growing challenge, and one function's priority may be another's back-burner item. Scheduling tasks and resources also becomes increasingly difficult as projects grow in number and complexity.

The solution for these companies is to develop a cross-functional capability, which characterizes Stage 2, project excellence. In this stage, companies take a project view of the world, aligning all functions for effective project execution throughout the product development process. Cross-functional and concurrent product development are the cornerstones of Stage 2.

Once project excellence is in place, the next step is to institute the same processes across all projects to reach Stage 3, or portfolio excellence. Portfolio excellence requires a common framework for product development across all projects. In Stage 3, companies are able to achieve platform leverage and portfolio balance, which are key to product strategy. Stage 3 processes allow management to select which projects to fund and which projects to cancel in order to achieve strategic alignment and to optimize the return on portfolio investment.

With portfolio excellence in place, the next step is Stage 4, co-development excellence. In this stage, companies are able to develop products with selected external partners using collaborative development processes. Besides boosting innovation and productivity, co-development allows companies to focus their resources on the areas that are strategically critical to the business.

BENCHMARKING PRODUCT DEVELOPMENT PERFORMANCE—WHY CURRENT METRICS PROGRAMS DON'T WORK

Using this Stages model, PRTM and PMG recently conducted a detailed benchmarking survey on the product development, pipeline, and portfolio management practices of about 40 companies across multiple industry segments. The study examined the relationship between overall product development capability, the maturity of portfolio management practices, and the level of product development performance as measured by a set of key performance indicators or metrics. Based on our analysis of the results, the most advanced companies in terms of pipeline management and overall product development capabilities (Stages 3 and 4), achieve:

  • Two times the industry average for return on investment
  • A 90% project success rate
  • 25% profit margins as compared to 12-18% for lesser performers
  • Faster growth overall

Benchmarking allows companies to objectively assess and quantify their product development performance, and compare that performance against others in the same industry. However, our study revealed that many companies struggle with which specific areas to measure, and how to get the most out of those metrics. In fact, our study found that even companies that have been measuring their performance for some time are failing to reap the expected benefits. Four root causes underlie the problem:

  • Performance metrics are not used to their full potential.
  • Metrics used are not the right ones—they're not consolidated or balanced. Companies focus too much on project and historical metrics, and not enough on portfolio metrics.
  • Tracking mechanisms are inadequate.
  • Bottom-line implications of performance improvement programs are not considered.

Performance metrics are not used to their full potential

We asked survey participants how they use performance metrics. As shown in Figure 1, over 70% of companies use metrics to review projects. But companies are not using metrics in a broader manner. Only 55% of the companies polled use metrics for planning and goal setting, 41% use them for benchmarking, and only 38% use them to link their strategy to individual goals. These numbers show that although most companies are tracking metrics, few companies are leveraging the full potential of metrics.

Reasons to Track Metrics

Figure 1: Reasons to Track Metrics

Metrics used are not the right ones

We asked respondents what types of metrics they were using [Table 1]. As shown in Figure 2, companies are collecting a wide variety of metrics, but only 35% of the respondents are systematically tracking consolidated/balanced scorecards. Over half (56%) of the companies are systematically tracking project metrics, but only 21% track technology metrics and only 26% track portfolio metrics. The implication is clear—companies are not getting a full, balanced picture of their product development performance. For example, companies that focus mainly on project-level metrics typically make sub-optimal product decisions, since portfolio and technology-level metrics are absent. A key reason for this imbalance is that most companies rely on a single function such as marketing or product management to measure product success (Stage 1) and have not achieved full Stage 2 Project Excellence capabilities, where management focuses on overall measures such as time-to-market and schedule slippage. Leading companies that operate at Stage 3 Portfolio Excellence have technology and portfolio metrics, such as overall project performance reported consistently across projects, which lets management focus on aggregate portfolio performance.

Types of Metrics Example/Definition
Consolidated/ Balanced Scorecard A comprehensive performance measurement technique that balances multiple performance dimensions within the PD process, such as market performance, financial performance, and resource management
Financial R&D spending, revenue from new products, return on investment (ROI)
Project Time to market, schedule/cost variance, design reuse
Functional Process Cost, volume, and cycle time for different operations
Product Market share, customer requirements coverage, profitability
Portfolio & Pipeline R&D throughput, resource/capacity load, portfolio balance of
R&D investment by project type and market segment, etc.
Technology Patent filings, technology readiness, commercialization success

Table 1: Example Metrics

Types of Metrics

Figure 2: Types of Metrics

Another issue we've identified is that companies lack predictive measures—those leading indicators of product performance. Historical metrics such as actual time to market are useful for measuring performance and analyzing trends, but predictive measures such as project complexity and project risk can help companies anticipate product development performance problems and take corrective actions. We asked survey participants to list the “pains” they associate with metrics tracking, and 80% named “lack of predictive measures” as an issue [Figure 3].

The data also showed that big or moderate issues for nearly 70% of respondents were “standardizing the data” and finding it “too time consuming” to track metrics (Figure 3). Companies without a standard set of appropriate metrics in place often struggle in these two areas. Without standardized—and therefore comparable—metrics, companies cannot interpret results and determine the appropriate actions to take.

Issues Associated with Metrics Tracking

Figure 3: Issues Associated with Metrics Tracking

Tracking mechanisms are inadequate

In the last few years we've observed that companies are using enterprise-wide systems to monitor product development performance. Development chain management (DCM) and product lifecycle management (PLM) systems automate metrics data collection and reporting. We asked survey participants how they collect and report metrics data. Only 24% of respondents are using enterprise-wide systems to track metrics [Figure 4]. Companies with enterprise-wide systems experience fewer problems such as difficulty in standardizing the data [Figure 5] than companies that haven't adopted the systems. Clearly, companies with support systems are benefiting from them, but the adoption rate of these systems is low overall.

Enterprise-Wide System Adoption

Figure 4: Enterprise-Wide System Adoption

Issues Comparison

Figure 5: Issues Comparison

Bottom-line implication of performance improvement programs are not considered

The survey showed that companies are making progress in using metrics programs as part of ongoing product development management. But although 68% of respondents [Figure 6] have incorporated metrics programs, only 47% [Figure 7] are measuring the financial impact of improvement programs. Without measuring the outcome of improvement efforts, companies have no way of knowing which initiatives have the highest impact.

Purpose of Metrics Programs

Figure 6: Purpose of Metrics Programs

Companies that Track the Bottom-Line Impact of Improvement Opportunities

Figure 7: Companies that Track the Bottom-Line Impact of Improvement Opportunities

PRTM APPROACH TO BUILDING EFFICIENT METRICS PROGRAMS

As noted earlier, companies with the most successful metrics programs take them seriously, and often implement them as part of a broader quality initiative. To make metrics work, a company should use a product development performance scorecard that contains a balanced mix of metrics in three key product development areas: 1) Project execution; 2) Portfolio and pipeline management; and 3) Product strategy and technology management.

Project execution

Based on our experience, we recommend that the following five key metrics be used to measure project execution: Time to market, schedule slippage, cost variance, time to profitability, and project performance to goal. The table below presents the definition and calculation of these metrics:

Metric Implication Calculation
Time to market How quickly projects are planned and executed Sum of the time spent from Phase 0 (Concept Phase) to Phase 4 (Market Release)
Schedule slippage How realistic planned project schedules are and how well you stick to those schedules   (Actual TTM-Planned TTM)
Planned Costs
Cost variance How realistic planned project costs are and how well project budgets are managed   (Actual Costs -Planned Costs)
Planned Costs
Time to profitability How quickly projects generate enough profits to recover development costs TTM + time from stable manufacturing to the point where all project costs are recovered
Project performance to goals How well projects perform relative to goals Average performance of the top five project goals, as ranked by the project team

Table 2

Portfolio & Pipeline Management

Greater efficiency in portfolio and pipeline management gives a competitive edge to Stage 3 companies. By regularly comparing the mix of products in their portfolio to that of others in their industry, Stage 3 companies can ensure that they have enough platforms and major projects to drive growth. They can also compare their R&D productivity against their total R&D spending.

Key portfolio and pipeline management metrics include new product revenue, R&D effectiveness index (RDEI), and pipeline throughput. Definitions of these metrics are summarized in the table below:

Metric Implication Calculation
New product revenue How much your newest products are contributing to your total sales Percentage of revenue from new products compared to total revenue, where “new” is defined as products that have not yet reached peak production
RDEI How the profit generated by new products compares to total R&D expenditures Profit from new products
R&D spending
Pipeline throughput How many major projects R&D can complete per million dollars of R&D spending # of major projects in Year X
$MM of R&D spend in Year X

Table 3

Product Strategy & Technology Management

Stage 3 and Stage 4 companies regularly review their product strategy to make sure it is aligned with their overall business strategy. They see technology management as a cross-functional process, and technology development projects as deserving the best practices of project execution excellence. Key metrics include strategic balance of released projects, technology readiness, technology leverage, and market renewal.

Metric Implication Calculation
Strategic balance of released projects The mix of platform, major, minor, and technology projects that R&D can release in a given year # of platform projects released total #of projects released (Same calculation applies for major, minor & technology projects)
Technology leverage How well R&D can leverage new technology developments into high revenue or growth products % of platform or major projects significantly enabled by technology development projects
Market renewal How much R&D spending is allocated to major revenue or growth opportunity projects (R&D spending on tech. developments + R&D spending on major product develop.)/ total R&D spending

Table 4

Implementing a Successful Metrics Program

Finally, to resolve the four root causes of metric program failure, and to build a metrics program that works, PRTM recommends the following 10-step approach:

Phase Step Objectives / Issues addressed
Phase 1: Define Detailed Improvements

1. Define metrics program

2. Define strategy and high level objectives

3. Define balanced performance metrics

4. Determine present process capability

- Define metrics program charter and work plan

- Define metrics program objectives that articulate how company will benefit from metrics -Issue 1 found in current performance measurement

- Link company's strategy and high level objectives to performance metrics

- Measure the right metrics issue 2 found in current performance measurement

- Assess the current metrics and leverage where possible

Phase 2: Implement

5. Define decision-making structure

6. Establish data collection and reporting process

7. Define metrics tracking systems

- Define who will review current performance to identify improvement opportunities

- Ensure timely, fact-based, and actionable decisions

- Define the tasks and responsibilities for data collection, analysis, and reporting to ensure efficient metrics tracking

- Ensure adequate tracking Issue 3 found in current performance measurement

Phase 3: Roll-out

8. Pilot metric process

9. Conduct ongoing performance reviews

10. Implement continuous improvement

- Identify first set of improvement targets and test the new performance measurement process

- Identify improvement opportunities and take corrective actions

- Revise and improve the metrics and the measurement process as needed to meet the changing business needs

- Track bottom-line impact of improvement program Issue 4 found in current performance measurement

Table 5: 10-Step Performance Measurement Approach

Advertisement

Advertisement

Related Content

  • Project Management Journal

    Getting Past the Editor's Desk member content locked

    By Klein, Gary | Müller, Ralf To reach acceptance, every research paper submitted to Project Management Journal® (PMJ) must pass several hurdles. This editorial aims to declare the editorial process and reveal major reasons for…

  • Project Management Journal

    Narratives of Project Risk Management member content locked

    By Green, Stuart D. | Dikmen, Irem The dominant narrative of project risk management pays homage to scientific rationality while conceptualizing risk as objective fact.

  • Project Management Journal

    Coordinating Lifesaving Product Development Projects with no Preestablished Organizational Governance Structure member content locked

    By Leme Barbosa, Ana Paula Paes | Figueiredo Facin, Ana Lucia | Sergio Salerno, Mario | Simões Freitas, Jonathan | Carelli Reis, Marina | Paz Lasmar, Tiago We employed a longitudinal, grounded theory approach to investigate the management of an innovative product developed in the context of a life-or-death global emergency.

  • Project Management Journal

    Investigating the Dynamics of Engineering Design Rework for a Complex Aircraft Development Project member content locked

    By Souza de Melo, Érika | Vieira, Darli | Bredillet, Christophe The purpose of this research is to evaluate the dynamics of EDR that negatively impacts the performance of complex PDPs and to suggest actions to overcome those problems.

  • Project Management Journal

    Navigating Tensions to Create Value member content locked

    By Farid, Parinaz | Waldorff, Susanne Boche This article employs institutional logics to explore the change program–organizational context interface, and investigates how program management actors navigate the interface to create value.

Advertisement