Of benchmarks and scorecards
reporting on multiple projects
There is a dark side to “divide and conquer.” At some point senior management will say, “One of the best pieces of advice for any project manager is Divide and conquer.” Take a complex, large deliverable, and break it into parts. Deliver it in small pieces, each of which can be managed separately, with as few interdependencies as possible. This strategy is an absolute requirement at times. Some efforts are so large that a single project manager cannot reasonably track and manage the entire effort alone. Some efforts are so complex that they need multiple managers, each with different specialized knowledge. Thank goodness for “divide and conquer.”
“But how are we doing overall?” The individual status reports from each subproject may not show the progress toward the overall goal. Cross-team problems may be ignored. Multiple formats from multiple authors can be confusing to read.
People want to see the forest, not just the trees. Creating an overall status report is a possible solution.
One approach to an overall status report is simple:
• Collect text descriptions of status and goals from the manager of each subproject
• Submit them all to a single person
• Concatenate them into a single status report, editing the result for consistent format and language.
This overall report can be extraordinarily helpful, to senior managers and subproject managers alike. Sometimes it is enough, but a project manager should have additional tools in his or her toolbox.
A second approach is to create an overall schedule for the effort. These schedules can be as simple as a milestone-only schedule, with key subproject deliverables tracked on a single plan. Dependencies and cross-team milestones can be represented. More complex versions will have a single master schedule that contains all tasks for all subprojects, including actual and estimated work hours. Scheduling software can help automate creating these master schedules. Creating schedules is often a tool-specific challenge, and in my experience there are easier ways to get clearer information to senior management.
Any project can use benchmarks and scorecards, but they are especially useful for projects composed of subprojects. Each subproject manager can create a custom scorecard containing critical, relevant data for their effort. Often a useful scorecard can be just a single printed page. Team members can use the scorecards to track progress, gauging concrete results against benchmark numbers. The overall project manager can then collect these scorecards, combine the relevant data from each, and produce an overall scorecard. Usually it does not capture all the metrics of all the subprojects, but it provides an accurate view of the forest of subprojects. Benchmarks can be combined to track overall project-level progress. The approach is satisfying because each manager has local control over his or her deliverable, and can set benchmark goals independently. Overall management can still track progress toward the ultimate goal with confidence and precision.
Based upon my experience managing software projects in financial services, this work is never simple or easy. Perhaps in a very mature organization with concrete, well-understood standards it would be. Most of us do not have the privilege of working in such an organization. Using a step-by-step process, though, anyone can establish some consistent standards for a multiple-project effort. If these standards survive the effort, the overall manager can take pride; he or she has increased the maturity of the organization while completing the project.
A note on terminology: Some organizations talk about “programs” composed of multiple “projects.” Other organizations use “program” to mean a whole product line, while still others talk about “projects” composed of multiple “programs.” This paper will avoid the term “program” entirely. Instead “subprojects” are parts of a “multiple-project effort” or an “overall project.”“Project manager” means any project manager, while “overall project manager” is someone running the overall project. “Subproject manager” is someone running a subproject.
Step 1: Determine Senior Management Needs
The target audience for these benchmarks and scorecards is almost always senior management. Most stakeholders are concerned with either day-to-day progress at a low level, or overall completion dates and cost. Management typically wants to see regular progress reports, to ensure that goals will be met.
A meeting is almost always the first step, to establish frequency of reporting, to gather a list of any key data they want to see, and to understand what questions they expect to have answered at each reporting period. At this stage, some reporting goals will be concrete, while some will be vague. Sample goals might be:
• “Weekly status reports by noon on Monday and a status meeting to discuss at 10 a.m. on Tuesday”
• “An email if anything seems off, otherwise a monthly meeting”
• “Whatever numbers seem most important to you at the time, and of course any changes to delivery date”
• “An up-to-date earned value graph, and control graphs for defects found and fixed”
Almost any combination of information is possible at this point. Of course the overall project manager will have some ideas about metrics that he or she wants to capture, but this meeting is about senior management needs. The project manager can collect other data, so long as management's needs are met.
Step 2: Plan Subproject Metrics
Each subproject needs metrics to contribute to the overall scorecard. The subproject should measure project results to show variance, problems, and progress. The subproject manager should come up with ideas about the metrics that he or she wants to collect. He or she may collect traditional project management metrics like earned value metrics, budget vs. actuals, and schedule slippage. Domain-specific metrics are often also useful, like number of defects opened or closed, feet of wire laid, number of rooms painted, number of code-reviews completed, and so on. The possibilities are infinite. The choice of metrics will depend upon the subproject manager's experience with metrics as well as the domain. Some subproject managers may even decide to collect no metrics of their own. At this point in the process, that is fine.
Step 3: Identify Common Metrics
Once the overall project manager has a list from each subproject manager, he or she can look for common metrics. Sometimes two managers will call the same metric two different things, or calculate the same basic metric two different ways. For instance, a straight budget calculation of actual costs to-date will usually be greater than an earned-value calculation of actual costs, due to differences in definition of earned and unearned costs. Note areas of possible conflict as well as areas of agreement.
At this stage it is also critical to compare the subproject manager's metrics against the senior management needs. The metrics list may meet some management needs completely, while other management needs have no supporting metrics. For the next step, it is critical to analyze the results and develop a strategy to meet management needs.
Step 4: Negotiate with Subproject Managers
With a list of common metrics, conflicting metrics, and missing metrics, it is time to talk to each subproject manager. There will always be some changes to negotiate at this phase. Changes at a subproject level might include:
• Changing data collection methods (i.e., use an earned-value budget or an accounting budget)
• Adding more data elements to the scorecard
• Renaming data elements
• Changing data collection/reporting tools to achieve consistent results
• Resolving differences in definitions (i.e., what does “50% complete” mean?).
The overall project manager may need to convince some subproject managers to begin creating a scorecard at all. It is not uncommon for people to manage projects without a scorecard, and the main benefit of a common scorecard is to the overall project manager and to the overall management team. Some people may resist the standardization required.
Any subproject manager who decided to collect no metrics in step 2 must now agree to collect a set of basic data, in order to participate in the overall project. These negotiations can be difficult. It is critical for the overall project manager to have a simple set of basic data needs, and to know exactly why each piece of data is necessary. Sometimes resistant managers fight metrics at every step of the way. The overall project manager should be prepared with advice or even practical help to begin data collection for each subproject.
Be aware that gaining their cooperation or acceptance at this phase is not enough. Sometime resistant managers can misreport or report late during the project execution, as a form of passive resistance. The overall project manager needs to understand each subproject manager; this negotiating period is a perfect time to get to know the overall project team really well. Wherever possible, try to generate goodwill and cooperation among the managers. Avoid creating situations where a single subproject manager appears to win every negotiation, while everyone else looses.
Step 5: Set Standards
The overall project manager takes the results of all the negotiations and creates a list of standards for the overall project scorecard. The standards should describe who collects what data and when. Responsibilities for each subproject should be clear and precise. Measurement methods should be exact. For instance, for “percent complete” for tasks, describe your method. Sample methods include:
• Either 0% or 100%—complete or not yet complete
• 0% if not started, 50% when started, 100% when complete
• (100* Act) / (Act + ETC)
• Different percentages for each phase of completion: 0% not started, 10% in-progress, 75% almost done, 90% ready for review, 100% passed review.
Clear standards are critical for consistent reporting. Even the best-intentioned managers will vary in the way that they report information. It is impossible to have 100% consistency for some types of metrics, but a good standards document increases consistency better than any other single tool.
While setting standards, beware of vague standards that are hard to enforce or audit. Ideally, metrics are like the results of scientific experiments: anyone else could use the same methods, observe the same phenomena, and get the same results. The overall project manager should be able to verify the subproject scorecards, at least within some range of certainty, like plus or minus 20%.
Networked tools for project scheduling, defect tracking, and work tracking simplify reporting considerably. Sometimes the subproject manager does not even need to report their data to the overall project manager. If the data is networked, the overall project manager can read and report on the data independently. This kind of transparency is a huge benefit in multiproject situations.
Step 6: Create an Overall Project Scorecard
With a standard document in hand, designing the scorecard is straightforward. A few key questions will drive formatting and layout:
• What information is most important?
• What order will we discuss this information at status meetings (if applicable)?
• How are we delivering the material (online, web, email, text-only, print, projector, etc.)?
• What are the limits of the physical media (color, font size, etc.)?
• Do we need to highlight any key information?
• Do any readers need the data in a particular format (graphical, percentage, table of results, etc.)?
• What formatting or aesthetic standards should we follow?
• Is this report too expensive to deliver?
In general, a good scorecard flows clearly from one set of information to the next, in the order that management wants to see and discuss it. Data is clear, formatting does not get in the way, and it is easy to scan the page for critical information. Totals are obvious and meaningful.
Time and time again, I have found that management appreciates a one-page scorecard. Sometimes this single page just has summary data. Additional pages can always show the detail. A single-page summary makes it easy to see project status at a glance.
Step 7: Create Overall Benchmarks
Certain scorecard metrics will be expected to steadily increase or decrease over time. These metrics are good candidates for benchmark-ing. Imagine benchmarks as a snapshot of expected values over time, with one expected value for each reporting period. The most common form of benchmarking is earned value analysis. Managers take the schedule and budget at the start of the project, and write the expected earned value at each period throughout the project. Regular status reports show the actual values compared to the benchmarks.
The overall project manager can use this technique for any metric on the scorecard. Benchmarks can be extraordinarily useful for schedule-constrained projects. Management can see with every report whether work is on schedule, ahead or schedule, or behind schedule, without reviewing hundreds of tasks.
Some advice for creating benchmarks:
• Only benchmark values that change predictably over time: number of open defects might seem like a good value to benchmark, because it must be zero at the end, but if the value goes up and down every week, it may be a poor choice.
• Benchmark a few values that really matter, that predict project success and show progress: too many benchmarks are confusing.
• Estimate likely progress: take into account vacation time, project phases, and seasonal issues, instead of just assuming a linear progression, like “two more each week.”
• Publish the benchmarks: benchmarks are a great motivational tool for many teams, encouraging team members to contribute to the overall goals.
• Use subproject benchmarks: give subproject managers control over their own benchmarks, and total their goals to set the overall project benchmarks.
The overall project manager must understand the purpose of the benchmarks. If there is a prize for the team at each met-or-exceeded benchmark, then set the benchmark goals high. Perhaps the manager sets the benchmarks so that he or she is only 30% sure that the team will make it each week. Perhaps the benchmarks show project completion several weeks ahead of schedule. Communicate these expectations to management and to the teams.
Sometimes benchmarks are a simple monitoring tool. The overall project manager should then set the benchmark at a spot where he or she expects the team to perform. Estimating uncertainty in the estimate is very valuable. Benchmarks can serve as a form of control graph, showing whether results are in control or out of control. The classic control graph has a single result expected over time, with horizontal bars for average, high, and low values. The benchmark graph would have an expected-result line that moves up or down over time, with control bars above and below it. When using this approach, uncertainty usually varies over time, so the uncertainty levels may change with each reporting period. As each goal comes closer in time, the likelihood of reaching the goal becomes more and more certain.
Subproject managers may or may not choose to create their own scorecards for each value. Over the life of a software project, one subproject might occasionally find and fix one or two defects, while another subproject finds and fixes 100 defects per week. A defect scorecard and benchmark could be useful for the overall project and the second subproject, but not for the first. Be flexible about applying metrics at the overall and subproject level.
Step 8: Senior Management Approval
When developing any reporting tool, it is critical to get approval of its target audience. A good project manager will involve the audience throughout the process. When and how to involve senior management always varies by organization, but with a draft scorecard developed and benchmarks set, it is essential to present a proposal to management.
This meeting is typically quite formal. The overall project manager distributes the benchmarks and scorecard ahead of time, and then reviews it step by step with management. A face-to-face meeting is recommended, even if the project communication calls for no face-to-face, regular status meetings. People need time to review and discuss the results. Different senior managers may need different information, and it helps them to hear each other defend each part of the report. Removing or changing a scorecard section requires everyone's approval, and often requires discussion.
Usually there is at least one change needed. Often the organization or format of the document requires changes. Sometimes the benchmark goals are too aggressive, or not aggressive enough. Occasionally the metrics do not meet management needs, and the scorecard goes back to the drawing board. Usually the overall proj-ect manager can note the changes in a follow-up email and begin using the new report for the next project-reporting period. Going through this process almost always produces a report that is better than the tools that came before it. Even if it is not perfect, management usually is ready to adopt it.
Step 9: Update Scorecards, Evaluate Actuals vs. Benchmarks
With each reporting period, metrics change. The overall project manager should set a reporting schedule with all subproject managers, to meet the reporting schedule required by management. Having everyone report all data as of the same date is critical to having a useful overall project scorecard. Assembling and reporting data for a complex project often takes several days. Even after the subproject managers have finished their reports, it may take a day for the overall project manager to file the overall report. A tightly organized team can improve that time, by meeting or beating reporting deadlines consistently. Data may be old by the time it is published; balance the competing needs for timely data against the needs for accurate, well-reviewed data.
Each subproject manager may have different reporting responsibilities for each reporting period. Typical responsibilities include:
• Producing the subproject scorecard and benchmarks
• Reviewing and approving data in shared databases
• Reporting additional data to the overall project manager, via email or phone
• Meeting with the overall project manager to review and confirm project results.
It is useful for the whole project management team to get together and review the overall results, before publishing them to senior management. This step ensures that no one is surprised by the results in the report. The overall project manager can reconcile any conflicts between subproject managers and can make last-minute corrections to the scorecards.
Benchmark values change every week. The overall project manager fills in the actuals for the week and compares them against the benchmarks. Any variances should have an explanation.
The overall project manager publishes the scorecards and benchmarks to management. The cycle repeats for every reporting period until the project is complete.
Step 10: Refine and Update
Projects are all unique endeavors, and an overall project is a collection of many unique endeavors. Even the best project manager will have trouble predicting the course of an overall project. Benchmarks are usually the first to change. Changes in staffing levels, unexpected project events (positive or negative), changes to scope, or changes to estimates all should trigger a review of benchmarks. Benchmarks loose their value as a management and motivation tool as soon as they become too easy or too difficult.
Often the project's metrics change as the project goes through different phases. The overall project manager should monitor overall project activities, to see when a particular metric is no longer relevant or when a new one should be captured. The overall project manager may choose to design a new scorecard for each phase or to keep a single scorecard, but remove and add data elements as the project evolves. Either approach can be successful. Adding new metrics mean that the overall project manager should consider adding a new benchmark goal. Dropping old metrics sometimes means dropping a benchmark.
This step goes hand-in-hand with updating scorecards. It may be triggered anytime after reporting begins on a project. Critical events in the project almost always trigger it, including changes to scope, budget, schedule, or phase.
During project closeout, the use of the scorecards and their evolution through the project often make their way into the lessons learned for the project. The history of scorecards and benchmarks will help the managers of future projects develop their own templates and standards.
Putting Scorecards and Benchmarks to Use
Following these 10 steps, a project manager can have an easier transition from managing stand-alone projects to managing projects that contain subprojects. These techniques are broadly applicable, useful for a wide range of industries. There are some key recommendations for people who are new to these tools:
1. Keep the metrics simple and easy to understand
2. Make sure that subproject metrics “scale up” to the overall project
3. Do not force metrics that work for one subproject onto all sub-projects; only use what works
4. Watch your math, because not all metrics sum up through simple addition; standard deviation adds using squares, for instance— see recommendation #1 to avoid this problem entirely
5. Explicitly state the uncertainty of the numbers at the start; it is next to impossible to collect data completely accurately or to forecast benchmarks perfectly
6. When benchmark values and actual values diverge, do not panic; figure out if the project is off-course or if the metrics are off-course and take corrective action
7. Never adjust the scorecard numbers in order to appear to meet goals; not only is it unethical behavior, it is increasing the projects’ list of problems and threatening the bond of trust with project sponsors
8. Do adjust metric calculations and benchmarks as needed; do so openly, with the consent of senior managers and subproject managers
9. Use the literature to select appropriate metrics; academics have studied and argued over appropriate metrics for decades, and it is expensive to ignore their advice
10. Only use sophisticated metrics after gaining experience with simpler metrics, and only if the audience of the reports understand the complex metrics thoroughly.
To summarize: FOCUS ON COMMUNICATION. These reports are a tool to communicate progress and status. Be creative, using color, graphs, charts, tables, pictures, or anything that helps communicate. An effective report will generate complete silence around a meeting-room table, followed by grunts of surprise, grunts of “humph,” exclamations of “wow!” and finally cries, “This is great,” or “NOW I understand.” The measure of effectiveness for a score-card or benchmark is its communication value. It is not enough to have numbers that accurately reflect project progress; they must also communicate that progress to the intended audience. Perhaps the ultimate compliment is when senior management starts telling your peers to use benchmarks and scorecards, and your peers come to you as the in-house expert for advice.
Ultimately the scorecards and benchmarks are about seeing the forest, despite all the trees. Managers who remain focused on communicating the state of the project “forest” will not get lost in the “trees” of numbers in their reports. Giving senior management the right level of vision, supported by accurate, demonstrable facts, is the ultimate goal of the exercise.
Proceedings of the Project Management Institute Annual Seminars & Symposium
October 3–10, 2002 • San Antonio, Texas, USA