It's been a long struggle. At last your project team can celebrate bringing up a new e-commerce website. Your team brought it in on time, even slightly under-budget. Your client loves it! But were you successful?
Do your project metrics assure that the site will be successful with its audience, an audience that might include:
• Business executives who comfortably multitask with phone, email, and printed memo
• Computer-wary users who avoid any hint of complexity
• Recent graduates accustomed to working with email, chat sessions, and instant messaging while listening to the latest MP3
• Eager-to-learn new users just discovering that the Internet is much more than AOL.
Now, imagine the challenge of crafting metrics from your available information to measure success. More to the point, will your organization as a whole benefit from what you've just delivered? Based on traditional project and management metrics, how can you tell? Something more is needed and that something is the topic of this paper.
As you read, a few explanations up-front will help the rest of your reading go more smoothly. One example is the Balanced Scorecard, a concept developed in a 1992 Harvard Business Review article (Norton & Kaplan, 1992), later expanded into the book The Balanced Scorecard (Norton & Kaplan, 1996). A Balanced Scorecard is a brief document translating a company's strategy into a comprehensive yet concise set of metrics focusing management's balanced attention on four areas: Financial, Customer, Internal Processes, and Growth and Learning.
Exhibit 1. Sample Product Development Stages
Another term is the awkward yet self-explanatory Internet-Based Product (IBP). These products are Internet services used by an organization to serve as a revenue-generating channel to deliver information, goods, or services to customers or business partners. While current Internet products are primarily web-based, IBP is broad enough to encompass existing Internet technologies (FTP, Chat, Instant Messaging, etc.) as well as future technologies.
In opening up new channels and methods, IBP's become gateways to a complex series of human and computer systems. Because of their closeness with the customer and the need to attract and hold the customer's attention, IBP's are best developed with project and product management methods in Internet-Based Product Development (IBPD) project teams.
The last term is framework, specifically a project management framework. Like the frame of a window, a framework serves as a guide to focus attention on what's important. For example, a Balanced Scorecard can be a framework for bringing project management, organizational strategy, and the voice of the customer together. No new product development project is exactly like any other, so using a framework assures flexibility while ensuring that what's important gets covered.
We will examine a Balance Scorecard Approach (BSA) and consider how to construct one for an IBPD project. The premise of this paper is that new product development is a natural candidate for the rigor of project management but requires some alterations to smooth its acceptance. While a BSA can be applied to the entire product life cycle, illustrated in Exhibit 1, we will focus on four stages: Go/No-Go Decision Gate, Development Project, Project Closure, and Operational Monitoring.
What's Wrong With Metrics?
Internet and IT Metrics
The Internet and IT worlds are rich with possible metrics. Hits, MIPS, uptime, bandwidth, function points, and others monitor internal Internet and IT processes. Because these internal metrics are not linked to business needs or strategy, they do not measure what is important for the business overall or for specific Internet-based products.
Once the financial measures of IT were obvious, as mainframes replaced numbers of employees. IBP financial metrics are now subtler, requiring measures other than financial ones. Typically, according to David Norton, investment in IT is a third-order financial effect as technology improves customer service, which increases customer confidence, which leads to increased sales (Norton & Kaplan, 2000).
In an IBPD, project technology is no longer the servant of product development but helps to define development processes. Technology issues affect return on investment, customer satisfaction, internal processes, and even the growth of employees—all the areas addressed by a Balanced Scorecard.
Management Metrics
One school of thought holds that it is better to measure anything than to measure nothing. But sometimes nothing is better than just anything. Wrong metrics can justify harmful decisions that lower morale, reduce productivity, and damage the organization.
In the absence of other metrics Gantt- or network-diagram project management tools are used and the time-budget ratio substitutes for more critical business metrics. It is important to be on time and on budget and these are common criteria for project managers. However, they say nothing about the success of the product in meeting the needs of the customers or building organization and processes for the future.
Financial Metrics
Financial metrics are universally used because numbers are the language of business. Focusing on financial measures guarantees that the project metrics are always in the past tense. That is, financial metrics always measure historical events. Business must be transacted, the cash collected and accounted for, before reports can be run. But the reports measure what happened upwards of 90 days ago.
This is using what you can easily measure, rather than what is important. We tend not to measure important things because the technology has changed faster than our tools and methods. The Internet has made historical data all but worthless. “Internet speed,” and the rate customers and businesses adapt to it, strips historical data of value.
Balanced Scorecards
A survey of metrics used in new product development conducted by the Goldsense Group indicates that metrics most useful to R&D professionals and executive managers are those that help shorten product-development cycles and tie R&D performance to profitability (Cabanis-Brewin 2002). These however, do not readily fit any of the metrics reviewed above.
This leads to a point Norton and Kaplan raise in their most recent book, The Strategy-Focused Organization: “The ability to execute strategy is more important than the quality of the strategy itself.” So they encourage the obvious answer: “Measure the Strategy”! (Norton & Kaplan, 2000). A Balanced Scorecard Approach (BSA) makes it easier to measure and track the distant, but related, effects.
The resolution of technology, management, and financial issues is firmly rooted in the decisions and actions of people—the project team. A Balanced Scorecard can give the team a tool to understand, anticipate, and resolve these issues.
Building a BSA for Internet products
IBPD is very much a creative act. The blend of creativity in programming and product development in the still-emerging medium of the Internet makes it very difficult to establish industrywide, or even organizationwide, standard metrics. Project participants, from executive sponsor thru the operations team, need to be observant, all-inclusive, and analytical when establishing the metrics for their BSA.
The key is to define an appropriate product or business strategy and ask, “How could I tell if I succeeded?” The answers will form your metrics. Building successful metrics requires that fundamental information, such as business and product strategy, market environment, organizational alignment, etc. are available to the IBPD team. Each function or department involved in the IBPD project must be measured by the same metrics, otherwise all balance is lost and costs will accrue instead of benefits.
As Kaplan and Norton point out, strategies are the unique and sustainable ways companies create value that, as a consequence, change relatively frequently. Not only have measurement tools not kept pace with these changes, but they also tend to focus on the tangible aspects of business. Knowledge workers more often create value via intangibles, such as with IBP, rather than by producing tangible goods (Norton & Kaplan, 2000).
The four perspectives a BSA uses to shift the focus are: Financial, Customer, Internal Processes, and Growth and Learning. Financial metrics, as discussed above, report historical events. The last three—Customer, Internal Processes and Growth and Learning—are leading indicators that reflect the intellectual and customer capital necessary for future growth of the organization.
Application to Four Stages
To illustrate how a BSA can provide meaningful metrics at various stages in an IBPD project, we will examine several common topics across several stages of a product development project. To simplify the paper, four of the stages shown on Exhibit 1 are used to represent a product life cycle:
Go/No-Go Stage Gate Project Closure
Development Process Operational Monitoring
These stages span the entire life cycle of a product but are spaced widely enough that differences between them are highlighted. For each stage these common topics will be briefly examined.
• Involved Roles—Taken from the roles shown on Exhibit 2 and showing which is Responsible for delivery
• Project/Product Deliverables—Using generic deliverables
• Locus of Control—Which metrics are most important to control this project at this stage
• Sample Metrics—Using generic metrics as examples
Go/No-Go Stage Gate
This stage follows the evaluation of the product's business case. At this point enough information should be available to determine the future business prospects for the idea and to decide either to continue with developing the idea (“Go”) or not (“No Go”).
• Involved Roles
• Project Manager—To manage the presentation and discussion ®
• Steering Committee—To make the decision
• Other Roles as appropriate to respond to questions, further the cause, etc.
• Project/Product Deliverables
• Product description/functional specifications
• Preliminary project plan
• Development budget
• Pro forma income statement
• Locus of Control
• This ought to be the most balanced of the stages in the BSA. While Financial metrics typically play the most important role at this Gate, Customer, Internal Process, and Growth and Learning metrics ought to receive equal consideration.
• Sample Metrics used to build the business case
• Financial—Return on capital employed and time to payment
• Customer—Customer acquisition cost and customer retention cost
• Internal Process—Product/Service quality, time, and price; and customer Service—on-time delivery or order error rates
• Growth and Learning—Staff skills inventory and business model analysis
Development Process
This is when the project team turns the idea into a viable business product or service. Typically the project team is the focal point of the work and the IBPD project ends with the “Go-Live” implementation of the website.
• Involved Roles
• Project Manager—Overall responsibility for the product development ®
• IT Development—Conducting the development and testing
• Product Manager—Frequent involvement in the iterative development and testing of the product
• Customer—Involved in developing the user interface and usability aspects of the product
• Users/QA/Customer/etc.—Involvement in system, usability, and acceptance testing
• Project/Product Deliverables
• Working version of the product
• Test results
• Installation and Operation documentation
• Locus of Control
• Typically the Internal Process and Financial metrics take center stage, with project management metrics dominating. However, Customer metrics must not be left behind. Typically, Growth and Learning metrics are de-emphasized since it is usually too late for training or skills gap analysis.
• Sample Metrics
• Financial—Project expense, product unit cost, and project schedule
• Customer— “Try holding your breath for as long as it takes your home page to load,” Perception of risk, and Improvements in user interface design
• Internal Process—Time to market (for new products or new releases of the IBP, EVA, CPI, etc. project management metrics) and actual vs. planned release date
Exhibit 2. Sample Project Roles
Project Closure
Once the website has gone live the IBPD project for that release is ready for closure. This will wrap up the project, compile the project documentation, and evaluate the project's success.
• Involved Roles
• Project Manager—To manage the closure and archiving of documents ®
• Steering Committee—To charter the closure and approve the archives
• Product Sponsor—To oversee the closure and maintain the product archive
• Project/Product Deliverables
• Project archives
• Closure approval documents
• Locus of Control
• The Financial and Internal Process metrics will be most important. The complied project documents should provide a wealth of information for these metrics.
• Sample Metrics
• Financial—EVA, CPI, and other project management metrics
• Internal Process—Number of new products or number of business ideas generated and percent of key-staff turnover
• Customer—Product Ease of Use
• Growth and Learning—Employee satisfaction (Learning and Growth)
Operational Monitoring
Actually the longest, and likely the most expensive, stage in the IBP's life cycle is daily operation. However, no successful website is implemented and then left alone. During this stage, metrics and their interpretation become critical indicators to the subtle enhancements each new release will receive.
• Involved Roles
• Product Manager—For the review, evaluation, and reporting of the server user logs ®
• IT Operations—For the daily operation and administration of the servers and network
• Business Operations—To monitor and analyze the back-office aspects of the site
• Project/Product Deliverables
• Daily operation and up-time of the IBP
• Regular analysis of server user logs (at least weekly)
• Features list of upcoming releases of the product
• Locus of Control
• Customer, Financial, and Internal Process metrics are equally important. Information is available for each on a daily basis. Each of the “Involved Roles” will be able to gather and analyze data.
• Sample Metrics
• Financial—Return on Investment, product unit cost, percent of sales by channel, gross margins for IBP's vs. typical channel, sales growth, order fulfillment costs, and time to payment
• Customer—Customer service improvement, amount of similar product available for free elsewhere on the Internet, customer navigation on the site, conversion of visitors to purchasers, time to delivery, customer acquisition cost, and customer retention cost
• Internal Process—IBP reliability (uptime and response time and accuracy), cost reduction in processes, order fulfillment time, cycle time for new IBP releases, percent of sale from new products, time to market for new products, and time to market for next generation of existing products
• Growth and Learning—Employee satisfaction, staff skills development, length of desired customer path for the IBP, and expansion of content management into functions.
Conclusions and Closure
A BSA is, quite simply, a focus on managing change by looking at the metrics that either need changing or reflect change in the larger environment. Project management is likewise a method of managing change while introducing that change into the greater environment.
There is a great deal of correlation between a BSA and current project management practices. For example, the role of a Project Office in developing PM standards and methods as well as their implementation is one that could easily support a BSA. For IBPD a specified project office could help develop a standard approach and could ease the implementation of a BSA (Block & Frame, 1998).
Here is how I believe the focus areas of a BSA would map to PMBOK® knowledge areas. This reinforces that a BSA can serve as a project management framework for IBPD, or other product development, projects.
| PMBOK® PM Knowledge Areas (PMI 1996) |
Balanced Scorecard Approach (BSA) |
| Project Integration Management Project Scope Management |
Internal Process Internal Process, Financial |
| Project Time Management Project Cost Management Project Quality Management |
Financial Financial Internal Process, Customer |
| Project Human Resources Management Project Communications Management |
Growth and Learning Growth and Learning, Customer, Internal Process |
| Project Risk Management | Internal Process, Growth and Learning, Financial |
| Project Procurement Management | Financial, Internal Process |
Points to Remember
In closing, there are a number of important points that will help your Balanced Scorecard succeed.
The Fewer the Metrics the Better
Complexity is the enemy of the BSA. Norton and Kaplan advocate no more than 20 broad measures enterprisewide for multibillion-dollar organizations, such as a Fortune 500 firm. For a single project or product fewer, more narrow metrics should suffice, but probably no more than eight.
Arthur Schneiderman, who developed one of the first BSA's at Analog Devices, advocates using relatively few metrics, saying the effectiveness of the BSA decreases as the number of metrics increases. At Analog Devices they tracked three areas using eight metrics:
• Customer Satisfaction (two metrics—delivery performance and product defect levels)
• New Product Development (two metrics—time to market and number of new product rollouts)
• Financial (four metrics—revenue, revenue growth, profit and return on assets)
Financial measures were only tracked by senior management. All other employees focused on the remaining two areas since they could have an effect on those results (Schneideman 2001).
Balanced vs. Unbalanced
While it is vital to keep the Scorecard as balanced as possible over the life of a product, each stage of the product life cycle requires a bit of re-balancing. As illustrated earlier, at times one of the four areas takes on more significance than the others. But, and this is critical, it takes on more significance, not more importance.
The difference is more than semantics. All four areas remain important throughout the product life cycle. At various stages in the life cycle you, and a product or project manager, will look to differing areas first. Not exclusively, just first.
Voice of the Customer
Using a BSA as a product/project management framework virtually guarantees you will hear the voice of the customer. That voice will come to you over the Internet as the written or verbal word and as action.
An IBP is an innovation to the business. But there are at least two forms of innovation going on at all times. One is the Growth and Learning of the business as it improves upon its IBPD processes and products. Just as important is the learning of its customers as they adapt to, and increase their expectations of, the IBP. Here the customer aspect of a BSA is crucial.
Communication
Maintaining a BSA will bring about improved communication as the functional groups within the organization, and the customers outside, interact more frequently and in more detail. The use of a limited set of comprehensive metrics helps by focusing everyone's communication on what is truly important.
The metrics of a BSA can easily be tracked by mapping the metrics to a Radar, or Spider, Chart. A polygon is drawn with spokes radiating from the center to a line representing a perfect score. There are as many spokes as there are metrics and each spoke is graduated from zero to perfection, typically 0 to 100. The performance against each metric is graphed onto each spoke. The result is a graph that shows areas of relative strength as well as areas in need of development.
Special Risk
A BSA can introduce significant risk into any IBPD project. A BSA starts breaking down all the cultural barriers that grow up between functional units and makes all results, not just financial or project-related, measurable and public. The risk: Too much change with too little time to prepare or respond.
Another risk, given the intense time-pressure of most IBPD projects, is that a high degree of persistence is often required to collect the metrics selected in the BSA. This is an excellent argument for minimizing the number of metrics tracked.