The mystery behind project management metrics

Abstract

To understand how to take data and transform it into metrics can be helpful to the project management practioner. Metrics on a project are important indicators as to the health of the project, and they provide details to make better decisions. The arrival of agile or Scrum has made the ability to gather and gauge project health even more challenging. This paper will describe an approach to taking data and making meaningful metrics by reviewing how metrics can be used, describing a ten-step process for gathering useful project metrics, then showing how the goal/question/metric approach can be utilized for determining traditional project metrics as well as better defining agile/scrum metrics.

Introduction

Do you need better metrics to determine project or organizational performance? Do you need better metrics to determine project team (agile or traditional) performance? How do you understand, evaluate, predict, and control data that can be transformed into information for better management of your project?

In this presentation we will uncover the mystery behind metrics. As a result of doing project metrics and helping projects find meaningful metrics, I have found the following three mysteries of project management metrics often come up. They are:

• Mystery 1. Where to start – Many project managers feel that current metrics on projects are not always enough. Often they don't know where to start.
• Mystery 2. Measuring the right goal – Are the current metrics providing what is needed on the project? Lots of data might exist, but how can you be sure you are measuring the right areas. This paper will describe a Goal/Question/Metric approach to uncover this mystery.
• Mystery 3. Agile/Scrum – Using agile or Scrum is new and often the traditional metrics do not always work. What metrics should be used?

The outline of the paper will begin with a foundation describing the differences between information and metrics.

A review of the ten-step process for metrics will be covered to help with the mystery of “where to start.”

The Goal/Question/Metric approach will be reviewed as a means to better define metrics and to assist in uncovering the “measuring the right goal” mystery. To determine some meaningful metrics for agile/Scrum, a review of traditional project management metrics and Scrum metrics will be reviewed using the Goal/Question/Metric approach based on the 12 agile principles.

Basics of Metrics

The main premise of this paper is to establish methods by which metrics can be used to help with business decisions. In a project, the project team is faced with decisions to be made daily. By adding the ability to have metrics, it can assist the project team in better making decisions pertaining to the project.

The second premise is that great project managers know their metrics and how key performance indicators on their project will be able to effect the outcome. Using good metrics can allow a project team to make any needed adjustments.

Let us first determine the differences between data, information, and metrics. Data is a collection of facts. An example would be the distance from Ontario, Oregon to Portland, Oregon is 307 miles. Information is meaningful data based or inferred from the facts of the data. Using our example, it is a long drive to Portland, Oregon from Ontario, Oregon based on the inference that it is 307 miles away. If our project is to deliver a load of wood to a distributor in Portland by 11:00p.m. and it is currently 6:00 p.m.

Metrics are the established relationships between the individual sets of data. Again from our example, at a speed of 75 mph, it will take roughly four hours to drive 307 miles, is a metric that combines relationships between the information and the data.

To better determine metrics, we need to measure and provide measurements for the metric. These metric measurements become an indicator that will allow the project manager the ability to adjust based on the indicator. In our example, the road sign of 307 miles was data that when made into a metric of 4 hours, the indicator provides insight into our project of driving to Portland. The metric became a good indicator as to the progress of our journey. It tells us that we should be able to make our schedule of arriving by 11:00 pm.

Of course on projects we will use other indicators based on metrics gathered. The project indicators allow a project manager to:

• Assess status of ongoing project – In our example, based on the 307 miles to Portland road sign data, we are on schedule.
• Track project risks – As we travel to Portland, we see data indicating snow clouds are forming. Using metrics of temperature and a weather radio, the probability of snow is high. This indicator could reduce the speed and effect our delivery time.
• Uncover problem areas – Data from the GPS map shows possible road construction delays ahead.
• Adjust tasks or workflow – As new data on our project appears, tasks will need to be adjusted. We just realized that the speed limit in Oregon is 65 mph. Also in Oregon, an attendant has to pump your gasoline. This can also create delays.

Many organizations get overwhelmed with the amount of data they can produce. To transform the data into information and then into metrics can provide better detail. In our example, we went from the raw data of 307 miles to knowing that if we can drive 75 mph we should arrive in approximately four hours. If our project consists of a task of delivering wood to Portland, Oregon, this metric can help in deciding if we now want to drive the four hours to Portland and how much time will be needed in the schedule.

Just as combining the metrics together for our journey to Portland, metrics can be found at different organizational levels in any company (see Exhibit A).

At the organizational level, some metrics might be:

• Performance against industry benchmarks
• Relative growth of organization
• Sequential growth of the organization

At the business level, some metrics might be:

• Performance of the department
• Efficiency of processes
• Customer satisfaction levels

In a project, the most common metrics are:

• Project progress against the schedule
• Cost incurred against budget
• Adherence to quality standards
• Risks versus reward

Exhibit A: Measures in an organization

As we transform data into metrics, metrics are used to understand, control, evaluate, and predict. The law of uncertainty states that the very act of measuring improves results.

The four aspects of measurement are (see Exhibit B):

• To understand or to gain a better understanding of an area of the project. This could be to better understand the baseline of the project to gain an understanding of the product.
• To evaluate or analyze metrics as part of the decision making could be areas such as how much can we compress the schedule if we add two more resources to the project. Evaluate also is used in analyzing and prioritizing risk.
• To control is usually associated with red flags or controlling quality. A contingency plan trigger for risk management is a good example of where metrics are used to control.
• To predict is the most common part of where metrics are used on a project. From predicting how much it will cost to how many resources are needed, the use of metrics is important on a project to predict.

Exhibit B: Four areas

Where to Start

Many project managers get flooded with data and information on their projects. Because each project is different, sometimes they don't know where to start and/or need to re-calibrate their metrics being used on a project. Linda Westfall wrote “The 12 Steps to Useful Software Metrics” to help software developers in determining some solid metrics for software development. The framework of her approach is used here, except the number of steps are reduced and some steps have been relabeled to better match project management terms. This ten-step approach is a starting point to help solve the “where to start” mystery.

1. Identify the project metrics customer. Establish who will be the customer for the metrics. On most projects the stakeholders are the customers. Often the customer is the project team, or even the project manager, who will use the metrics to make decision or use the metrics to modify a plan.

   To help with the definition of the customer, lets review the objectives of metrics:

   • Communicate to senior management

   • Educate staff

   • Establish the current state of the project

   • Value to the business

   • Forecast

   • Predict problems

   I use two sets of metrics: one to manage with (I viewed myself as the customer), and one to communicate out to others (my stakeholders that are my customer).

2. Identify project goals being measured. This pertains to what the goals of the metrics are. A model of defining the goal, selecting possible questions to define the goal, and then finding what metrics can be used to answer the questions. The next section covers the Goal/Question/Metric in more detail.
3. Define project metrics. A metrics requirement statement is useful in determining the purpose of the metric. (See Exhibit C.)
   

   Exhibit C: Metrics requirement statement

   To predict the number of days left on the project in order to plan the release date is often a metric used on a project.

4. Identify project data to collect. Using what is called “counting criteria” is important to remember when identifying the data to collect. It helps everyone interpret the data in the same way. Some examples of counting criteria are: hours, days, and money.
5. Define project data collection methods. Three keys to collecting data are:

   • What to collect?

   • Who should collect it?

   • How to collect?

   After collecting the data, the next question is: can it be automated? The ability to take time to automate and pull metrics is often worth the effort.

6. Project data analysis. Once the metrics are gathered, being able to do anaylsis is where decsions and action can begin. An example will be: what thresholds (established boundaries), when crossed, indicate and action is needed. The simple example of “if the projected cost is over, then escalate” shows a threshold. Some metrics that can help identify the trigger are:

   • Cost variance

   • Schedule variance

7. “Define project reporting and feedback” is determining how to present the metrics. Three challenges are:

   • Report format

   • Timing or how often

   • Methods of delivery

   I have seen many great reports and metrics that suffer from information overload. To present metrics and information in an easily digestable way is an art. On a large program intitative or project management office, a communication specialist is recommended to help convey the information.

8. “Document the project metrics process” is being able to replicate or audit how the metrics were produced. Three challenges in this area are:

   • Mapping the process to be repeated or audited

   • Getting it documented

   • Consistent implementation

   Often great metrics are produced and presented. Be prepared to show where the data came from and if any alteration occurred. A good executive will take one metric and drill into it to validate the source. If the source the executive is asking about is suspect or unknown, the rest of the metrics in her mind could be flawed.

9. Human aspects of metrics. It is important to remember that measures affect people and people affect measures. It is important to measure the correct metrics to help people behave accordingly.
10. Continuously improve the project management metrics process. Look for ways to streamline and improve. Remember to throw out any metric no longer needed, no matter how cool.

The ten-step process for gathering project management metrics is a starting point for producing metrics. We will return to step two of the ten-step process and use the Goal/Question/Metric method to better understand the right metric to match the goal.

Measuring the Right Goal

To assist with measuring the right goal or checking to see if the correct metric is being used, the Goal/Question/Metric approach is effective. The end result is a metrics requirement statement that ensures a well-defined metrics based on goals. Other advantages are:

• Eliminating misunderstandings
• Communicating needs
• Providing a basis for metrics design

The Goal/Question/Metric Apprach starts off with: What is the goal? After you identify the goal, list possible questions that need answering to determine if the goal is being met.

Then select metrics that can provide information to answer the questions. “The Goal Question Metric (GQM) approach is based upon the assumption that for an organization to measure in a purposeful way it must first specify the goals for itself and its projects, then it must trace those goals to the data that are intended to define those goals operationally, and finally provide a framework for interpreting the data with respect to the stated goals” (Basili, Year, p. 2).

The goals are matrixed to questions and then to metrics that can help understand, control, evaluate, or predict the questions regarding the goal. (See Exhibit D.)

Exhibit D: Project goal question metric

If the goal is to deliver on time and within budget, the questions might be: How long will the project take?—a metric of effort estimates and schedule estimates. A metric of cost estimates would help with the question: How much will the project cost?

The following are additional examples of the Goal/Question/Metric to show how it can be used on a project:

• For project management metrics, see Exhibit E
• For estimation and planning, see Exhibit F
• For risk management, see Exhibit G

Exhibit E: Project management metrics

Exhibit F: Estimation and planning

Exhibit G: Risks

Dangers of Metrics

The dangers of metrics is they can be used in making bad decisions.. Now that you have metrics, be aware of the dangers that metrics may bring. Often project managers become like airplane pilots that use their instruments to fly when they need to look out the window of the plane to see where they are going. Often project managers use metrics for decisions based on what the metrics are telling them whereas by collaborating as a team, a better decision could have been made. Metrics often become justification for making bad decisions.

In the book Leaders Eat Last, Simon Sinek compares the differences between small numbers and big numbers. He starts off by quoting Joseph Stalin: “The death of one man is a tragedy. The death of a million is a statistic”(Sinek, 2012, Loc 1895). Sinek continues, “It is one thing for numbers to represent money or products. But when big numbers start representing human beings, as Stalin told us, our ability to empathize starts to falter.…But a decision using a spreadsheet to lay off four thousand people at some large corporation loses tangibility and becomes something that just needs to be done to meet certain goals” (Sinek, 2012, Loc 1928). The need to use metrics accordingly is important to remember.

The other danger of metrics is drowning in the sea of information. Metrics often can become so important that too much time is taken from getting the real work done. The old adage of “spending all our time reporting on the nothing we are doing because we are spending all our time reporting” can become a possible hazard.

Agile and Scrum

Armed with the ability to better define metrics, many agile project managers struggle with the ability to use traditional project management methods in methodologies such as Scrum or agile variants. Using the 12 agile principles as published by the Scrum Alliance and the goal/question/metric approach can find meaningful metrics for the agile project manager.

In the article “Measuring Alignment with the Principles of Agile” published on the Scrum Alliance website, Madhu Venantius Laulin Expedith described some Scrum metrics. I have added additional metrics to this list based on my experience as a ScrumMaster. Each of the 12 agile principles will be used as the goal, and questions will be generated to assist in finding some metrics that could help in measuring the project and/or the team.

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. (See exhibit H.)

   Goals:

   • Customer satisfaction

   • Product owner satisfaction

   • Early and continuous delivery

   Questions:

   • How satifisfied is the customer?

   • How will product owner be satisfied?

   • What value are we delivering?

   Metrics

   • ROI

   • Customer satisfacton survey

   • Net promoter score

   • Product owner feedback

   

   Exhibit H: Scrum GQM

2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

   Goals

   • Changes to increase the competitive advantage

   • Late changes are good

   Questions

   • Are the changes bringing business value?

   • Do the changing requirements increase the ability to compete?

   Metrics

   • Requirements stability (# of backlog items)

   • # of changes added to backlog by sprint/iteration

   • Cost of change / estimated business value

   • Cost of change / estimated increase in sales over the competition

3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

   Goals

   • Reduce time for working software

   • Deliver software often

   Questions

   • How many weeks before working software?

   • How many features in first release?

   Metrics

   • % top features in backlog in first release

   • Release burn down

   • Milestone analysis

   • % time spent on defects that cause delays to delivery

4. Business people and developers must work together daily throughout the project.

   Goals

   • Get developers and business working together

   • Daily interaction on the team(business and developers)

   Questions

   • How engaged is the business?

   • Are daily scums or stand-ups held?

   Metrics

   • Percent of effort logged by the business

   • Percent of effeort logged by the developers

   • Integration points with other projects missed

   • % of daily scrums with all team members (including business) attending

5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

   Goals

   • Provide environment and support

   • Trust team

   Questions

   • Do we have the right environment for the team to be effective?

   • Are team members happy with the work?

   • Does the team trust each other?

   Metrics

   • Percentage of nontechnical impediments

   • Employee satisfaction survey/scorecard

   • % of teams with sprint goals and measurements

6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

   Goals

   • Have face-to-face meetings

   Questions

   • How often does the project hold face-to-face meetings?

   • Does the development team know the business requirements?

   Metrics

   • Percentage of team distribution by location

   • Percentage of defects with lack of communication as the cause

   • % with product owner

   • Percentage of impediments related to communication

   • Number of communication channels

7. Working software is the primary measure of progress.

   Goals

   • Working software

   Questions

   • Are we delivering working software?

   • How well is the software selling?

   • How many support calls are from defects?

   Metrics

   • Delivery efficiency

   • Velocity

   • Quality objectives met (% defects)

   • Net promoter score

   • % market share

8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

   Goals

   • Maintain pace or cadence

   Questions

   • Are we getting better from sprint to sprint?

   • Are improvements reviewed and implemented each sprint?

   Metrics

   • Resource utilization rate

   • Acceleration from sprint to sprint

   • Reductions in impediments

   • % retrospective improvements implemented

9. Continuous attention to technical excellence and good design enhances agility.

   Goals

   • Technical excellence and good design

   Questions

   • What % of technical debt to get a product out the door?

   • Is the design improving?

   Metrics

   • Percentage of defects with design-related defect causes

   • Defect density

   • Effort required to refactor

   • Design quality index

   • Technical debt

   • Total cost of ownership

10. Simplicity—the art of maximizing the amount of work not done—is essential.

    Goals

    • Reduce work that is not essential

    Questions

    • Are we eliminating user stories?

    • How efficient is the team?

    • How many defects or rework?

    Metrics

    • Percentage of open defects in the product backlog

    • Percentage technical debt in the product backlog

    • Release burn-down

    • % of stories no longer needed

11. The best architectures, requirements, and designs emerge from self-organizing teams.

    Goals

    • Better architecture

    • Better requirements

    Questions

    • Are we holding retrospectives?

    • How often does the team swarm?

    Metrics

    • List of impediments due to team organizing issues

    • Sprint retrospective

    • % architectural standard creating technical debt

12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

    Goals

    • Team reflection

    • Team adjustments

    Questions

    • How well do we trust each other?

    • How often are non-performers or detractors removed from the team?

    • Are retrospectives being held?

    Metrics

    • Retrospective effectiveness – % of top items implemented

    • Retrospective efficiency

    • Increased team feedback

    • % of team members trusting each other

Success Factors of Project Management Metrics

Some best practices that can assist in utilizing project management metrics are:

• Project metrics dashboards to show team status and progress
• Give diligence to the “soft factors” or things that are hard to measure
• Project metrics can only show problems and provide ideas on how to improve
• Action takes place as a result of analyzing the data the results bring
• The goal is improvement through measurement, analysis, and feedback, not measurement and metrics collection alone.

Conclusion

Project management metrics can be a mystery. By learning more about the basics of metrics and how metrics can be used to better understand, control, evaluate and predict project outcomes. Just as scientist uncover mysteries by learning more about a mystery, the project manager by applying the tem step process and the Goal/Question/Metric method can solve the project management metrics mysteries.