Every organization asks questions such as: “Are we achieving the results we desire?” “Are we meeting the objectives of our projects?” “Are we meeting customer success criteria?” And, “Are we achieving our desired return on investment?” Success is contingent on being able to make predictions and meet commitments relative to products and services. With projects increasingly a way of life, it also is important to demonstrate the added value to the organization through project management methodologies, tools, and techniques. A metrics program is necessary. Metrics can help identify important events and trends that can separate problems and opportunities and help guide organizations to informed decisions. How do you know that your work in project management truly is making a difference and contributing to organizational success?
It has long been recognized that sound measurement practices are integral to basic management activities such as project planning, monitoring, and control, and metrics are ever present throughout project management. As organizations mature in terms of project management and more organizations follow a managing by projects approach, management that is objective and performance based relies heavily on measurement practices. Metrics need to be integrated into project life-cycle processes to support decision-making, project selection, and portfolio management and to guide product and process improvement. Metrics can serve as indicators of organizational project management maturity. Metrics help understand capabilities so achievable plans for producing and delivering products and services can be developed. They also enable people to identify important events and trends and to help separate problems and opportunities. This can help provide better control of costs and schedules, reduce risks, improve quality, and ensure that objectives can be achieved.
Organizations, however, often use metrics incorrectly. They may not collect any metrics, may collect a lot of metrics but have no valuable use of the data that are collected, or may collect data but use only a small part of the information that can be gleaned from the metric. Some metrics only are collected to fulfill a corporate requirement to report on information by certain dates. In some organizations, measurement is an end unto itself. In terms of project management, often, those at the middle, who have achieved project management certification and are users of specific project management tools and techniques, recognize the value of project management, but this value may not be fully embraced by those at the top. A metrics program is one way to communicate effectively the benefits of project management to senior executives.
This paper discusses actual experience in implementing a project management metrics program at First USA. It describes the need for the metrics program, the reasons we focused on metrics, and specific metrics we considered. It also presents some guidelines for effective implementation of a metrics program.
Planning the Metrics Program
One of the dangers is that there are potentially so many things to measure that one can become overwhelmed by opportunities. It is important to plan a metrics program by identifying project and process management issues, selecting and defining the corresponding metrics, and integrating them into existing processes.
Definition and Purpose of Metrics
A review of the literature on metrics showed there were many definitions of this term. Metrics, as used in this paper, are small measurable quantities that have a predictive or measurement capability; they can offer a glimpse of the status of an item relative to expectations. A good metric is one that contributes to the right decisions in a timely way based on fact rather than feeling (Augustine, 1999). Metrics can identify important events and trends in the organization and can help guide the organization toward informed decisions. They can serve as the basis for clear, objective communication with project stakeholders. Metrics also can be used to show where an organization stands in terms of project management maturity. Metrics are a basic tool used to monitor and aid in determining the health of processes. Metrics can be used to measure the status of activities, take a process view, and gauge the contribution of project management to the organization (Dymond, 1995).
The Need for a Metrics Program at First USA
In the Technical Services group at First USA, the projects that we manage are primarily infrastructure, not application development. Our environment is fast-paced and very dynamic, with numerous projects occurring simultaneously. Each project manager is responsible for a number of projects. We realized first that we needed to structure our processes in managing projects. If the processes were followed consistently, as determined through measurement, we felt we should see an increase in project successes. We also felt that we needed to provide a long-term strategy to grow the skills of the project management group. These objectives led us on a search for an industry maturity assessment methodology for projects and automated tools that would work in a non-application development area. The challenge proved to be greater than we imagined as most of the methodologies and tools on the market were targeted for application development efforts. However, we firmly believed that an industry standard in addition to our internal standards was critical. Maturity assessments for projects even in non-application development teams are crucial as all projects affect the success of the business. From this assessment, then, goals and targets for measurement can be derived (Pulford, 1996).
Assessing the Environment
We decided on an industry project management maturity model that detailed processes that we could apply in our infrastructure environment. Next, we conducted our project management maturity assessment manually on two levels. This was completed through an analysis of our methodologies, processes, tools, and techniques through detailed surveys of personnel at various levels. The first assessment covered the maturity model practices, while the second assessment evaluated First USA project management procedures as mandated by our Program Office. The assessments were completed using data and deliverables from active and completed projects and customer surveys. Performing two assessments determined how we measured against the industry and how we measured against our internal processes. The assessments helped us better realize our strengths and weaknesses, and more importantly, they pinpointed specific areas for improvement based on our own priorities that were incorporated into our assessment.
Determining Specific Characteristics of Metrics
Concurrently with our assessment, we determined specific characteristics of metrics that we tied to specific maturity levels in the assessment methodology. These characteristics were: performance, stability, compliance, capability, and improvement (Florac, 1997).
Performance Metrics
Metrics in this category emphasize performance to show the ability to deliver products and services with the qualities, timeliness, and costs that customers require. When performance metrics vary erratically and unpredictably over time, the project management process is not in control. With performance metrics, it is possible to assess repeatability of project management techniques and determine whether or not internal and external requirements are being met. These metrics are not judgmental and basically gather specific facts. Their focus is on attributes of quality, quantity, cost, and time.
Stability Metrics
Stability is central to each organization's ability to produce products and deliver services according to plan and to improve processes with better and more competitive products and services as the end result. For example, if people in the organization are using the project management methodology, but products and services still are not meeting the customer's requirements, the methodology is not stable and must be improved to reduce the variability. However, for project management to be stable and predictable, it also must be used consistently in the organization.
Compliance Metrics
Compliance means that the project management standards of knowledge and practice exist and are followed. It assesses adherence to the process; fitness and use of people, tools, technology, and procedures; and fitness and use of support systems and organizational factors, such as management support. Project management processes that are clearly defined, effectively supported, faithfully executed, reinforced, and maintained point to project management maturity and are more likely to be stable, repeatable processes. For this to be the case, project personnel must be aware of, trained in, and given the tools needed to best execute these processes. Mature project management depends on consistent execution of the process. The methodology must be executed as defined. For example, although an organization may have a Program Management Office, the latest project management software, a methodology, project mentors, and a project management training program and career path, there is no guarantee that people in the organization actually use the project management concepts, tools, and techniques. The issue to be examined is the penetration of project management practices across an organization. To what extent are project management practices being used? Measurements involving compliance do not address performance per se. Their purpose is to provide information to help explain performance results, variations, and patterns that are observed. Metrics can detect and diagnose what actually is under way.
Capability Metrics
Capability is a necessary characteristic to see if performance satisfied customer requirements and whether it meets business needs; any variations then need to fall within ranges required for business success. Project management should have predictable results. If so, the project management processes are termed capable. Analysis of performance then can identify areas where the capability of a process can be improved to better support business objectives. Capability may need to be improved to satisfy competitive pressures or to comply with customer needs. It helps to predict the quality of products.
Improvement Metrics
Improvement metrics focus on the performance of the project management process. How can project management help move the organization to a level of greater profits? What are ways to determine if project management is working successfully throughout the organization, and if the changes that have been introduced are effective? In order to promote improvements, people must understand the business goals and strategies of the organization and also the priorities, risks, and issues associated with these goals and strategies. Improvement metrics thus need to correspond with business indicators. The costs and benefits of project management improvement need to relate to the business indicators used in the organization. Determine whether the changes proposed can reasonably be expected to have the effects predicted. After changes are implemented, examine results.
Selecting Metrics
We identified specific, meaningful metrics that could help us reach the next level of maturity and also could help us enhance our capabilities in project management. Rather than just counting possible defects, though, we wanted to emphasize metrics as a way for a knowledge management system for our project management practice. The metrics we selected were ones we could use to see if there was a pattern across the various projects we managed within Technical Services. The purpose was not just to supply data for reports but instead to provide data with predictive, future-oriented value (Dixon, 2000). We also emphasized performance, stability, and compliance metrics as we felt most of the capability and improvement metrics were ones that were more appropriate for focus at the entire organization level, were already collected at our corporate level, or were outside the purview of the individual project manager.
Performance Metrics
In the area of performance metrics, we considered techniques for completing a project on time using appropriate resources, techniques to complete a project within budget, realistic expectations for project goals and objectives, the importance of recognizing both our customers and our own tolerance for risk and errors, and the necessity for our performance as project managers to meet our customers’ needs. We felt it was important to determine whether each project followed processes of scheduling, costing, reporting, monitoring, and controlling, and whether or not risks were identified throughout the project before they became problems. Were the First USA standard project management procedures actually used? We also wanted to be able to obtain a view of the performance of all the projects that were managed by Technical Services. Representative metrics we considered in this area were:
• Number of key milestones completed and number of key milestones missed. This can be done by baselining the project schedule and tracking milestones to the WBS identification number on the project. Differences in the scheduled production date and actual production date can be tracked. This information can be used to determine why dates were missed so preventive action could be taken on future, similar projects and can help in the preparation of schedule templates.
• Completeness of requirements. It is important to determine whether requirements are defined sufficiently and in terms so they can be met at the beginning of the project. Using a document that defines the scope for the project, you then can track the number of requirement changes on this document. This will enable you to see whether these changes were used to more succinctly define requirements as the project ensues, or whether the changes resulted because the requirements were not defined in enough detail at project initiation.
• Testing compliance. Using a Test Plan form, you can show expected results and actual results. You can measure testing compliance as a way to develop a knowledge management database as to why tests fail so preventive action could be taken.
• Accuracy of Cost Estimates. Using a Cost Estimating form, you can record the project costs during the resource planning process. Then, you can track your actual budget and records of expenditures based on actual costs at the work package level to this estimate throughout the project. This can help in improving the quality of future estimates on similar or related projects to better establish a realistic budget at the beginning.
• Unexpected requests for information. Determine whether your customers or executive managers requested additional information on each project, or whether your existing reports were useful ones.
• Contractor Performance. Working in conjunction with your Procurement Department, track each contractor's progress in meeting schedule, cost, and technical performance goals.
• Risk impact. Using a risk planning form, you can build a link to specific work packages in the WBS. This will help you determine the number of risks identified, risks that actually occurred, and their impact. These data can be used to help you determine the types of risks that may be expected on certain types of projects at certain phases in the life cycle.
• Amount of effort spent on project management practice improvement. This metric can tell if the effort is increasing, decreasing, or staying about the same. This enables you to show that improvement is not add-on work and that that process improvement can be done without placing too much strain on your resources. Improvement should not be an overwhelming demand on people's time that has a beginning and an end but instead needs to be a normal way of working.
Stability Metrics
For stability metrics we considered items to show our ability to complete projects according to plan and also metrics that could help us improve the processes we used for project management. Representative metrics we considered included:
• Effectiveness of schedule and cost tracking and controlling process with the value of the scheduling and costing tools and techniques in managing projects. Each organization typically has various tools and techniques in use in this area ranging from Microsoft Word files or Excel spreadsheets to advanced project management scheduling software used to handle multiple projects. Assess the usefulness of these tools to see if people need training in advanced features of the product, and to see whether additional reports should be prepared of data in the system.
• Types of problems that required escalation concerning working relationships, authority, and responsibility issues. Prepare a project charter to show the project manager's responsibility and authority that is signed off by both your customer and project sponsor. Then, use this charter as the basis for any authority and responsibility conflicts that may occur. At the end of each project, issues in this area should be reviewed to see the types of problems or conflicts that required escalation.
• Effectiveness of the contract change management system. Working with the Procurement Department, establish a method to track specific change requests and the change request process. See the types of analyses that need to be conducted and their effectiveness based on the impact on the specific project. See the levels of necessary signoffs for various types of changes.
• Revisions to the subsidiary plans of the overall project plan, such as the procurement, cost, or quality management plan. Determine the number of revisions needed to each of these plans based on the actual project execution. Also, assess specific stakeholders that contributed to the development of these plans in the planning phase of the project. Use these data to see if existing templates used for planning are sufficient, or whether any changes are needed, and also to see if other stakeholders need to be involved in your planning process.
Compliance Metrics
For compliance metrics, we considered use of existing tools and techniques, adequacy of support systems, and adherence to corporate level processes and procedures, with a focus on why processes may not be performing as they should or to see if insights were being obtained. Representative metrics we considered included:
• Applicability of methodology for the range of projects under way. Collate existing templates, forms and the methodology in a short handbook containing the approach to use including forms and plans to complete. Make it web-based for training new staff and for reference for existing project managers so everyone knows the requirements on each project.
• Product conformance with requirements. A statement of detailed project requirements needs to be prepared on each project with signoff by the project manager and customer. The number of change requests per project needs to be tracked to assess the completeness of the requirements analysis and definition process. Over time, change requests should decrease.
• Customer acceptance of project deliverables. Obtain official customer signoff on project deliverables, and at the end of each project, send a survey to each project customer. This survey can be used to rate performance and the overall success and satisfaction with the project deliverables in terms of project requirements.
Implementing the Metrics Program
Some things that are easy to measure often are not particularly important. Others that are more difficult to measure tend to be the most important. Measurements must be well defined, and all involved must accept and support guidelines for their use. The project managers needed to be involved and supportive of the program for success since they are the participants in the process and the source of the data that are gathered. Our emphasis was to use metrics for results and further project management improvements.
After reviewing the project management assessments and the recommendations, we formulated a Continuous Improvement Program (CIP). Its intent was to describe our strategy for the next two years, aligned with First USA business goals, and how we would accomplish it. We wanted to track achievement toward critical corporate objectives. Prior to the development of the CIP, we wanted to be sure that all the project managers on our team knew, understood, and supported the project processes and were following them.
The project managers participated in a one-day workshop, where they flowed many of the project processes. We facilitated several sessions over the new few weeks reviewing each process with all the project managers to make sure each one was helpful and understood. This way each person became part of the process so resistance to change was minimized. Our goals included improving project performance, stability, and compliance as well as increasing project manager skills to achieve better project results. Our two CIP objectives were to create a biannual project assessment process and establish metrics to ascertain project management value.
If we were going to assess projects twice a year, then we needed to have a process to support it. The process was detailed to show the months when we would complete the project evaluations, analyze the results, and tie the information into Bank-mandated performance reviews. This process would allow us to redirect project manager efforts if needed based on objective data. The first manual assessments we completed were resource intensive. As we expect to have more projects in the subsequent years, we identified a project management tool to use that would automate the assessment. This tool further enabled us to survey our customers at the end of each project.
We recognized that common definitions, processes, and reporting approaches are needed so everyone can become familiar with the information collected, and that there also needs to be consistent measurement for all the projects that are under way. Furthermore, for the metrics to be useful, they should satisfy three criteria:
(1) Communication—can others understand what is being measured and what is included and excluded; can people understand how data are collected so they interpret results correctly?
(2) Repeatability—could someone else be able to perform the measurement and get the same result?
(3) Traceability—are the origins of the data identified in terms of sequence, activity, status, environment, tools used, etc., in order that overall project management process performance can be improved?
Metrics need to apply across projects and span the project management life cycle. We wanted to treat the metrics effort as a flexible process, not just a prescribed list of data to collect and reports to complete.
We prepared a data collection plan with our team to support collection of reliable data. Since change is threatening at any level, our team needed to buy-into the program to see that the data collection exercise was a useful one with data collection the prerequisite for the analysis that would follow. Our project management metrics also had to be integrated into our existing processes so they were not considered just another request for information. We recognized that if measurement and analysis were made part of our existing processes, they would become integral to our procedures on all levels. Rather than being “laid over” as an afterthought, effective measures would be built into our project management practice so the most useful data are generated (Dixon, 2000). The methods to capture data had to be as easy as possible. Buy-in for the measurement activities was needed. Every team member needed to know why each metric was collected to understand its importance so the data collection effort is clear, concise, and consistently repeatable. Most importantly, we needed to demonstrate to all levels that the metrics were to be used to help us improve our capabilities.
Some of the metrics were focused on completing deliverables for a life cycle phase such as a signed design document. Others allowed us to compare known versus unknown risks on projects. For example, using the risk information, we could determine if a particular category of projects, such as building new servers, is more susceptible to a higher rate of unknown risks. If this is true, then we will examine and improve the processes around that category of project. By continuing to measure risks after process improvements have been made, we should see a decrease in unknown risks with the projects running more efficiently for the Bank. In summary, improvements are based on metrics and are not subjective opinions or conjecture.
Metrics must be well defined, and guidelines for their use must be accepted. Metrics need to be viewed not as a one-time exercise but as a continuing initiative toward enhanced productivity and effectiveness. They can serve as a benchmark for evaluations of future performance. The project management metrics program is the basis for actions to support project management improvements. A valid measurement, after all, is worth more than one thousand guesses! Clearly understood factual data facilitate correct analysis and help ensure there is agreement on what is happening and what should be happening. Measurements can, and do, form the basis of clear, objective communication with project stakeholders
