New directions and innovation in metrics-based management
Parviz Rad, PhD, PMP, Principal, Project Management Excellence
The organizations that follow a management-by-projects approach develop a performance-based style and continually become more sophisticated in the area of project management. Such a project management philosophy is highly formalized, is carefully methodical, and relies heavily on project measurement practices that are objective and consistent. Project management metrics are a key element in this process by focusing on portfolio management, program management, progress management, process improvement, and benchmarking. The selected focus of a metrics system, which in turn defines the structure of the measurement activities, is guided by organizational strategic objectives. Metrics also can serve as historical-based benchmarks for evaluations of future performance, with the hopeful expectation that they will facilitate continuous improvements.
This paper describes the purpose and importance of metrics to project management and presents a description of metrics in three categories: those involving project things, those involving project people, and those involving the enterprise. This paper describes processes and procedures that an organization must embrace as it establishes a culture focused on metrics and friendly toward projects. The guidelines presented here can by used by project professionals in their own organizations to set the stage for such a metrics-focused culture..
The Importance of Metrics to Project Management
Dinsmore and Cooke-Davies (2006) note that the rate of project success has not kept up with the tremendous growth of the project management profession. In fact, they state: “A majority of organizational projects fail to deliver even half the benefits they were designed to provide.” (p. 1). They further describe that the primary function of metrics, and information systems, is control as a comprehensive suite of metrics can provide people with the information they need to help ensure project and organizational success and note that the “word metric is used to describe a form of measure for communicating information in a compact and meaningful way” (p. 186). Rose (2006) that “Metrics are a means of measurement for determining the degree of conformance to specifications.” (p. 54). The Project Management Institute states that “A metric is a measurement of something.” (p. 16). It further explains that while tangible items may be able to be measured directly, such as error count, other items are more intangible, such as customer satisfaction, and therefore, must be made tangible through surveys or other types of instruments.
It has long been recognized that sound measurement practices are integral to basic project management activities such as project planning, monitoring, and control. Metrics are ever present throughout all phases and facets of project management. As more organizations adopt a management-by-projects approach, there is an increasingly heavy reliance on measurement practices that facilitate informed decision making, using indices that guide product and process improvement. Metrics need to be integrated into project life-cycle processes to support project selection, project execution, workforce improvement, and portfolio management. They should be used in data collection as well as in predictive models. Consistent tools and procedures, used by competent personnel, lead to lower overall project costs and increased corporate projects.
Metrics can provide indicators of organizational project management maturity, growth, and sophistication. They also can quantify the adequacy of the organization’s vision, the quality of each project deliverable, and the progress of projects, programs, processes, and products. As Augustine and Schroeder (1999) note, a good metrics system is one that contributes to the right system in a timely manner based on fact rather than on feeling. The most effective programs are ones in which the metrics are tailored to the organization’s important issues and strategic goals.
In the context of project management activities, metrics can offer a glimpse into the full range of expectations of a particular deliverable item or the tasks that would produce that deliverable. Metrics also can gauge the status of a deliverable item relative to expectations for scope, cost, and delivery date as they are aimed at small, measurable attributes that have a predictive or comparative capability.
Notably, metrics by themselves do not impart any value to the organization because they do little, if anything, to resolve the issue being monitored. Metrics do not make decisions; people do. Metrics simply provide the foundation and rationale for decisions. Thus, the return on investment of a metrics system derives from the actions that project professionals take to manage the issue at hand. With metrics, important project decisions can be made on an informed basis. Metrics can bring objectivity to the tools for monitoring project progress and can help advance the organization by promoting uniformity, accuracy, and repeatability.
Three Key Categories of Metrics
Traditional “Things” Metrics
The quantitative facets of projects, or project “things” are visible and tangible signs of project implementation and ultimate project success. Until the 1990s, most project management metrics programs focused on the “things” aspect of the project – primarily those involving schedule, cost, and resource use. This approach still predominates in those organizations that have not advanced beyond the second level of a staged project management maturity model or the “standardized” process in the Project Management Institute’s Organizational Project Management Maturity Model (OPM3®).
“Things” metrics characterize tools that assess the progress and success of projects almost as if the project processes exist without the intervention of people. They quantify and measure cost, schedule, scope, quality, and the project’s overall success in meeting client needs. Many of these metrics relate to the monitoring and controlling phase of the project, even though foundations for them should be established during the planning phase, and focus on progress in terms of the quality of deliverables, including variances in cost and schedule.
These metrics can be used to assess tradeoff decisions between the areas of scope, time, and cost. For example, should the project’s schedule be compressed in response to a desired earlier delivery date by the client and if so, what is its impact on resource requirements and the budget? Using cost estimating data, for instance, a metric can be established to record the initial cost estimate and updated estimates as the project moves through its life cycle; this metric can help improve the quality of future estimates on similar or related projects. Additionally, a metric can be devised to determine the effectiveness of the processes to be used to control schedule and cost. Other examples of project “things” metrics include:
- Assessment of client requirements. Requirements may be described as the articulation of a client’s wishes and typically are documented in the project’s scope statement. Differences between planned and developed requirements should be tracked by assessing the status of each component of the deliverable as it moves through the life-cycle activities. Metrics for requirement volatility can be prepared by maintaining a detailed history of the direct and indirect requirements changes and the rationale for each one. Metrics also can track activities conducted to make changes in the deliverable in order to resolve any physical or performance defects. The number of changes to project scope, the desired deliverable quality, assumptions, and constraints can form the basis for a suite of metrics. As an indirect measure of overall project success, it is possible to devise a suite of metrics for the timeliness in resolving ambiguities of the project scope and for the number of times scope issues had to be escalated to executive management for resolution.
- Schedule metrics. Schedules can be developed for the tasks that comprise the work package level of the Work Breakdown Structure (WBS) or for any of the WBS intermediate levels. Although the WBS is not a metric per se, it can be used to effectively enhance estimates and schedules. A detailed schedule network also is not a metric in and of itself but does provide the foundation for metrics to evaluate the pace and productivity of the project. Schedule metrics can be based on the most detailed activities of the project or on its aggregate components.
- Resource and cash flow forecasts. Developing detailed project estimates typically begins at the lowest level of the WBS. Each level of the WBS should be tagged with the total resources required, grouped by resource category. At level zero of the WBS, the project’s total cost is determined. Combining the predictions for resource expenditure with resource cost and sequencing schedule activities provides a wealth of project information. The combination of these indicators provides a resource demand forecast and a cash flow forecast. Then, based on these results and also with knowledge of organization’s constraints for cash flow and resource availability, project plans can be adjusted by reducing resource demand or making organizational adjustments in favor of project progress.
- Project quality. Literature (Rad, 2002) shows that volatility of project scope and quality is the leading cause of implementation errors and eventually serve as the leading cause of variance from the project management plan. This means that the success or failure of scope and quality issues tends to overshadow project performance in other areas. To achieve an in-depth understanding of client involvement and satisfaction, each client could complete a survey at the end of the project to address project performance in terms of requirements, the success in meeting these requirements, and also the client’s general satisfaction with the project deliverables.
- Risks. It is inevitable that each project will have a certain number of associated risks. Risk metrics thus can be divided into metrics that categorize the nature of risk events, those that categorize the probability of occurrence of these events, and ones to predict the impact of the risk event on the project’s outcomes.
Additionally, the “things” metrics that are especially useful to the project manager, and possibly to the project team, tend to include operational issues, such as sophistication of processes, effectiveness of processes, and ultimately the impact of those processes on the project deliverable. These indices allow all team members to maintain more effective control over costs and schedules by reducing risks, recognizing opportunities, and improving quality, thus achieving significant project objectives to the satisfaction of the client.
Organizations must use the sum total of all of their resources to sustain success. As Lynch and Cross note (1995), a motivated team can successfully discharge continuous improvement duties in areas including client satisfaction, flexibility, and productivity. Furthermore, through the movement toward management-by-projects and greater use of virtual teams, project work is becoming more people centric than location centric. This means that team-related issues, such as team building, teamwork, team organization, team turnover, and team experience are far more critical to successful project performance (Sanvido, Grobler, Parfitt, & Guvenis 1992). Efficiency and quality can be maximized by optimal allocation of people on projects who have the right attitude, mindset, and motivation for the specific environment. Therefore, it is somewhat surprising that many organizations do not include people factors in defining project metrics. Performance metrics within the context of people issues can identify opportunities for motivating desired behaviors in team members. By definition, those organizations with higher levels of maturity will have metrics that are dedicated to people-related issues at both the project and the enterprise levels.
People metrics, or ones that characterize the behavioral attributes of people, do not lend themselves to easy quantification. They can include indicators of procedures for conflict management, communication, collaboration, teamwork, and technical competency. They also can deal with the features of the enterprise environment that promote leadership, integrity, and professional responsibility. They are intended primarily to evaluate, directly or indirectly, whether team members are executing their tasks well. People metrics can be viewed as ones that measure the friendliness of the organization toward the team and the friendliness of team members toward one another.
Since people are both participants in the metrics program, as well as sources of its data, people metrics cannot have the desired benefits unless their development recognizes this basic fact. Project professionals must be involved in defining metrics they feel are valuable and should be able to recommend starting or stopping the use of certain metrics as they deem appropriate.
Because project success depends on effective team performance and the collective performance of the team depends on the individual performance and behavioral traits of each team member, each team member must understand the overall team dynamics as well as his or her own individual objectives. Team members’ roles and responsibilities must be defined in terms of the nature of the individual results needed to support the team’s work processes. A formalized people metrics system provides the logical basis to respond to questions such as:
- What attributes of work are considered quantitative, and what attributes are considered qualitative?
- Is a qualitative approach is followed for metrics, how can performance be monitored to ensure that it meets or exceeds expectations?
- Are different types of metrics systems required for co-located teams and virtual teams?
- How can appraisal systems become team friendly?
- What combination of metrics is required for organizational success?
- How can an appraisal system become team oriented rather than individual oriented?
Using instruments that address these issues, a team member can obtain a more detailed appreciation of his or her technical performance and general behavior. This information can help team members understand how other team members react to everyday project situations. The intent is not to acquire a tool to change people’s behavior, but the goal is to obtain a clearer picture of the behavioral attributes of team members, recognizing that personal challenges and characteristics of a team environment can become sources of frustration and stress that can impact team performance. A set of metrics with accompanying guidelines can show team members how best to cope with these different behaviors and use the inevitable team conflicts innovatively and progressively. People metrics, therefore, can address team member interrelationships with one another and focus on teamwork issues, such as trust, collaboration, competency, communication and conflict.
Project management maturity additionally can be regarded as both as a people attribute and as an organizational attribute. If maturity metrics are used for people purposes, the spirit of measurement is how much they improve the progress of a given person or given project. The project management maturity of an organization is directly and predictably related to the success of the collective projects of the organization, which in turn is related to the success of individuals. At the upper end of the maturity spectrum is the organization that consistently completes its projects ahead of time, under budget, and exceeding quality expectations. Such an organization can boast about its cadre of motivated, competent people who are the primary reason for its project success.
Enterprise metrics deal with the structure and environment of projects, particularly as influenced by policies and procedures. They indicate whether the environment is project friendly, the involvement of project teams in the organization’s strategies, and the recognition of the project management discipline in the organization and show whether an organization is gaining a competitive advantage by achieving its strategic plan. To acquire and sustain a competitive advantage, organizations must be informed about all aspects of the project and the organization as a whole. Critical information includes the current state, and the future direction, of the enterprise and the industry.
Since project management has been recognized as a critical and competitive tool of world-class companies that are achieving desired business performance (Thamhain 1996), a metric can be established to determine whether the effort associated with improvements in the project management practice is increasing, decreasing, or staying about the same. This metric would quantify whether the culture of the organization is permeating the organization, and whether project management practices are effective. If a formalized maturity evaluation is conducted, the results will further establish a baseline for improvements in the success rate of projects. In terms of portfolio management, these metrics may include progress toward established targets, the value and benefit realization of the portfolio as a whole, as well as aggregate measures which include strategic goal achievement, stakeholder satisfaction, asset management and development, risk profile, and resource capability (PMI, 2006). Metrics thus can be regarded as enablers of enterprise project management.
A suite of metrics can be used to determine the existence of realistic, effective project management policies and procedures. For example, a serious distinction should be made between having unused procedures on the shelf and using existing procedures on a regular basis. The metrics associated with a maturity model can be used to signal how often and how regularly project personnel follow established processes, which in turn will shed light on the effectiveness of prevailing project management policies and procedures. The indicators of the maturity model further highlight the sophistication of key practices in the organization, particularly those in need of improvement.
At higher levels of maturity, metrics evolve with organizational process enhancements until the target process is mature and performs at a consistently high level of effectiveness and efficiency. Changes to processes are typically small and infrequent.
Because the Enterprise Project Management Office (EPMO) typically is the cornerstone and focal point of the enterprise’s project management initiatives and is actively and directly involved in the development of project management processes as a means of improving project delivery throughout the organization, the effectiveness of the EPMO should be evaluated by the degree to which the organization’s portfolio of projects is successful in terms of responsiveness, cost, and duration. Once the EPMO is fully established, it is reasonable to expect that the success rate of projects will increase. Metrics should show that more projects are being completed successfully and have fewer overruns. The overall cost of maintaining the EPMO further should decrease over time. Its usefulness and effectiveness also can be evaluated by indices for the existence, and ease of use, of a standard project management methodology. Other items to consider include: integration of a standard project management information system with the other systems in the organization, incorporation of lessons learned into the project management methodology, improvement in project predictability, comparative performance relative to other organizations that are considered to be best in class, and continuous increases in project management competencies.
Guidelines to Follow to Implement a Culture Focused on Metrics
Implementing a metrics system can help facilitate improvements in project success measures such as scope, quality, cost, schedule, and client satisfaction. However, while many project professionals believe that enlightened organizations must implement a full complement of project life-cycle metrics in all branches of their organization where project accomplishment is part of the strategic mission, the goals of the metrics system will vary in urgency as determined by the strategic objectives of the organization. The metric system’s sophistication, and its funding, also can vary depending on whether the overall goals are to improve individual project performance, divisional project performance, enterprise project performance, or enterprise project management maturity. The metrics system should be regarded as a tool to show sustained organizational benefit achieved primarily because of a project management culture. The underlying objective of a project metrics system is to provide accurate, verifiable information that can be used for improved performance of the enterprise’s project management capability.
Sometimes even if the metrics system is aligned with company strategy, reporting of performance information is fragmented and isolated within specialty areas. This pattern of reporting can then lead to mixed results on overall performance. For example, the quality department may measure and report quality attributes at a certain time period, the finance department may collect and report the values of cost attributes at a different time period, and operations may report daily on project delivery metrics. In other cases, the metrics designed to measure the performance of the units in the functional areas might not support the overall intended organizational strategies. Activities that are cross-functional, such as those performed by an EPMO, may be overlooked as links may not be in place between project performance and operational performance.
The cost of failures resulting from the lack of appropriate guidelines far outweighs the efforts required to establish the guidelines. A customized metrics system established for the enterprise could be characterized as one that is:
- Driven by business goals
- Consistent across the organization
- Adaptable to specific user needs
- Focused on collecting data as directly as possible
- Compliant with operational data collection policies
- Supported by all affected parties, including the client, suppliers, and valued partners
A number of key guidelines are suggested here. First, the goals and success criteria for the metrics system should be established. This is necessary because there are so many potential aspects of projects and project management that can be measured. It may be that the organization may wish to collect metrics only to fulfill a corporate requirement to report on information on certain arbitrarily targeted dates, or on the other hand, the organization may wish to use metrics to help achieve higher levels of corporate profits. Other goals include refining the estimating processes for cost and duration, improving the project delivery process, handling unusual and unplanned project events more effectively, enhancing communications, improving responsiveness, and achieving greater client satisfaction. Determine what needs to be measured and follow a principle advocated by Collins (2001) as he notes that “Stop doing” lists are more important than “to do” lists (p. 143).
Additionally, through an effective metrics system, each project can become an opportunity for learning that benefits the entire organization. Success criteria for the metrics system, and for project management, thus must be developed; this means that the metrics system’s impact on the following must be identified: individual projects, programs, portfolios, morale improvement, and operational benefits.
In planning a metrics program, it is important to identify the most important project and process management issues, select and define the corresponding metrics, and integrate these metrics incrementally into existing processes. A detailed implementation plan is suggested, with procedures and policies to use the results derived from the system. Each person’s involvement in the metrics program should be known and communicated. Communication about metrics initiatives must be direct, consistent, and widespread. The metrics system implementation project should be treated like a project, perhaps even more so, to highlight the advantages of effective planning and monitoring.
The main purpose of using a single metric or suite of metrics as part of a comprehensive model must be determined before the related data are collected. The direct or indirect linkage among the metric, the project goals, and the organizational goals must be highlighted. The thresholds of the metric that define normalcy should be clearly defined along with values for exceptional performance and unacceptable performance. The metrics system, therefore, should respond to the following questions:
- What is the attribute or the subject of the metric?
- Is the right attribute being measured?
- How often is the measurement performed?
- What are the thresholds of the metric?
- Who are the people who need to be informed of a specific metric’s value?
- How can we motivate the team with this metric?
The system should be designed with an eye toward the collection of data for both project and enterprise success factors. It may be advisable to first focus on the things attributes of one project. Then, data collection and reporting can be extended to people attributes of the project and eventually to things and people attributes of more than one project. Ultimately, the program can be expanded to include enterprise attributes that encompass all projects.
The implementation plan should specify incremental progress milestones such as first visible impact, intermediate significant achievements, and completion target. The metrics system’s initial focus should be on areas that involve the greatest impediments to improved project results. Staff from all stakeholder organizations should actively participate in the implementation. Greater project personnel involvement will foster greater commitment, increased sharing of lessons learned, improved coordination, and early warning of problems. The implementation team should conduct reviews of the metrics system’s effectiveness in the light of continuous improvement initiatives. Then, the implementation team must communicate the results and progress to all stakeholders.
Metrics should not be a one-time, data collection exercise, but they should be viewed as part of a continuing initiative toward enhanced productivity and effectiveness. The focus of developing and using a metrics system is not to provide a quick fix for projects in trouble but instead to establish the path for a journey to maturity and then to excellence. While project professionals have often stated, “What gets measured, gets done”, others (Rowan, 2005), further state: “What gets measured, gets better.” A properly-designed metrics program can ensure that the organization continues to pursue the right projects even when organizational and environmental changes occur on a day-to-day basis during the life of these projects.
Much of this paper is adapted from: Rad, P.F. and Levin, G. (2006). Metrics for Project Management Formalized Approaches. Management Concepts, Vienna, VA.
Augustine, T. & Schroeder, C. (June 1999). “An Effective Metrics process Model.” CrossTalk. 12(6), 4-7.
Collins, J. (2001) Good to Great”. New York: HarperCollins Publishers Inc.
Dinsmore, P.C. & Cooke-Davies, T.J. (2006). The Right Projects Done Right! From Business Strategy to Successful Project Management. San Francisco. Jossey-Bass.
Lynch, R. L., & Cross, K. (1995). Measure Up! Yardsticks for Continuous Improvement. 2nd Edition. Cambridge: Blackwell Publishers.
Project Management Institute (2003). Organizational Project Management Maturity Model (OPM3®). Newtown Square, PA: Project Management Institute.
Project Management Institute (2006). The Standard for Portfolio Management. Newtown Square, PA: Project Management Institute.
Rad, P.F. (2002). Project Estimating and Cost Management. Vienna, VA: Management Concepts.
Rose, K.H. (2005) Project Quality Management Why, What and How. Boca Raton, FL: J. Ross Publishing, Inc.
Rowan, L. (2005). What Gets Measured Gets Better: The Application of Learning Metrics. Chief Learning Officer. Retrieved 07/09/06 from: http://www.clomedia.com/content/templates/clo_article.asp?articleid=935&zoneid=13.
Sanvido, V., Grobler, F. Parfitt, K. & Guvenis, M. (1992) “Critical Success Factors for Construction Projects.” ASCE Journal of Construction Engineering and Management. 118(1) 94-111.
Thamhaim, H.J. (1996, December). “Best Practices for Controlling Technology-Based Projects. Project Management Journal, 27(4) pp. 37-48.
© 2006, Ginger Levin and Parviz Rad
Originally published as a part of 2006 PMI Global Congress Proceedings – Seattle Washington