Constantly Doing the Right Projects and Doing Them Right

What Metrics Do You Need?

University of Technology, Sydney; University College, London

Abstract

Corporate strategy is implemented through projects, so if organizations want to improve the execution of their strategy, they need to improve the success of their projects. To do this, they need to improve their corporate project management capability, of which metrics are an essential component. This paper provides a research-based answer to the question, “What are the characteristics of a suite of project management metrics that is capable of supporting an organization's efforts to improve project success?”

But before answering that question, the connection between corporate strategy and project success is reviewed, and the current role of project management metrics is examined, along with some commonly encountered problems.

Strategy, projects and project success.

In today's turbulent times, it seems as if every organisation holds on to some form of “strategic intent” as a compass to guide it through the rough waters. This may be a collective intention arrived at through a deliberate process of analysis and dialogue that is then formally written down in an “organisational strategy” document or it may represent the emergent characteristics of the combined actions of the whole organisation expressed as a purpose or a series of goals. But just as an individual person seems to need a meaning in order to make sense of his or her life (Frankl, 1959), so an organisation seems to need a strategic intent in order to survive in a turbulent business environment. (deGeus, 1997)

The strategic intent, of course, varies from organisation to organisation, but it will always encompass two kinds of goals: improvement of the current products or services and the processes and technologies for delivering them; and innovation and introduction of new products or services, processes or technologies. (Anbari, 2003) Regardless of industry or market sector, each of these two kinds of goals requires projects to be established, executed and completed so, in a very real sense, projects can be considered as an integral part of “strategy in action”, and the successful implementation of every strategy can be said to depend, to a greater or lesser extent, on the successful completion of projects.

Unfortunately, at this point in the argument, it all becomes rather more problematical! Behind the discussion of project success lies a seething mass of complex assumptions and inter-related concepts that have led one author almost despairingly to ask, “Measuring success – can it really be done, and if carried out, what purpose does it serve?” (De Witt, 1988, p164). Difficulties abound for many reasons: the different viewpoints, interests and expectations of groups of stakeholders involved in any given project; the subjective nature of perceptions of “success”; the tendency of perceptions to evolve over extended periods of time; the difficulty of assessing complex phenomena using simple metrics – the list is a lengthy one. On closer examination, project “success” turns out to be a rather slippery subject!

And just as an exact definition of what constitutes project “success” eludes us, so the results achieved by projects continue to disappoint those associated with their sponsorship, execution or implementation (Morris & Hough, 1987; O'Conner & Reinsborough, 1992; KPMG, 1997; Cooke-Davies, 2001).

It has been argued elsewhere that one way through the maze of discussions about project success is to consider the answers to three separate questions: “What factors lead to project management success?” “What factors lead to a successful project?”, and “What factors lead to consistently successful projects?” (Cooke-Davies, 2003, p185). But the focus of that article is success factors, those inputs to the management system that lead directly to the success of the project. These should not be confused with success criteria, which are the measures against which the success or failure of a project will be judged. Since this paper proposes answers to a question about “metrics” (the science of measuring), the focus will be rather more on success criteria than on success factors, but it is necessary to spell out the distinctive characteristics of the criteria that are normally used to assess the success of projects at each of the three levels.

Level 1. Project management success – was the project done right?

This is the measure of success that has dominated the practitioner-oriented literature on project management. In the folklore of the project manager it is about managing time, cost and quality. In reality, project objectives are rarely this simple. There will often be a business case to be born in mind or a gross profit to be made; there may be health, safety and environmental objectives to be accomplished; if the project is a technical one, or a “platform” new product development, there could be scientific or technical goals to reach.

Nevertheless, the principle is simple: once the objectives of a project have been clearly defined and the constraints spelled out, then the project manager, and her or his team, can use their best endeavours to deliver the project so that it meets the objectives within the constraints. If anything changes, which is likely given the inherent uncertainty that is involved in any new endeavour, then techniques such as project risk management and project change control can be called into play as appropriate. As a guided missile seeks its target, adjusting its trajectory as appropriate along the way, so the project team seeks to achieve the project objectives. Is this then an appropriate level at which to measure the success of a project? There are three different kinds of arguments that suggest that it is.

Firstly, modern project management has developed from a base of managing relatively “discrete” projects, each with its own organization, and each established to accomplish specific purposes (Morris, 1994). The kind of success criteria that are broadly used as measures of “project management success” have not only been those most commonly applied in the history of project management (PMI, 2000), but they also allow the project team as a coherent organizational unit to be accountable for its own performance and the practice of aligning accountability with authority is one of the well-attested principles of good management practice.

Secondly, the underlying concept behind measuring success at this level is based on the well-understood principles of first-order cybernetics (Schwaninger, 1997) in much the same way that a thermostat or a guided missile operate. This is clearly appropriate for projects in which both the goals and the methods of achieving them are relatively clear at the outset (Turner & Cochrane, 1993).

Thirdly, the capture of data about the extent to which projects within the same enterprise are successful in terms of project management success enables the enterprise to compare and contrast the practices that are generally associated with successful projects with those associated with unsuccessful ones. This in turn provides the enterprise with valuable information about which project management practices are in need of improvement within project teams.

These are convincing arguments that support the case for continuing to measure project management success for many projects in many organizations. It is far from being the whole story, however, and for the second level of success it is necessary to turn to the second of De Wit's levels – what he calls “project success” (1988).

Level 2. Project success – was the right project done?

This level of project success is perhaps the one that is of most interest to the “owner” or “sponsor” of the project. It is about “value for money” in its broadest sense. The assumption is that the project will be successful only if it is successfully operated, and delivers the benefits that were envisaged by the people and organisations (i.e. the stakeholders) that agreed to undertake the project in the first place. An analysis of six recent project management “bodies of knowledge” identified sixty core elements that are central to the way a project manager thinks about his or her work (Cooke-Davies, 2001). When these are clustered into eleven topic areas, and related to each other through a “systemigram”, it becomes clear that “anticipated benefits” become the touchstone not only for formal “stage gate” reviews of projects, but also for the continuous “informal assessment” of the likely success of projects carried out by owners, sponsors or senior management, and influencing decisions about priorities and resource allocation.

In spite of the relative silence of the project management profession on the subject of “benefits management”, there are three reasons why this is an appropriate level at which to measure the success of a project separately from the first level that was discussed, project management success.

Firstly, as Exhibit 1 shows, benefits are not delivered or realized by the project manager and project team, they require the actions of operations management. This calls for a close co-operation between the project team on the one hand and the “sponsor,” “customer” and/or “user(s)” on the other and involves dialogue with a wider cross-section of the organization than is necessary for project management success. Secondly, delivering project success is necessarily more difficult because it inevitably involves “second order control” (both goals and methods liable to change) and thus brings into play an additional set of corporate processes to those that are involved in delivering project management success. And thirdly, the extent of project success is unlikely to become clear during the life of the project itself, whether success is measured quantitatively in terms of financial benefits or qualitatively in some less tangible form. For these three reasons, project success is itself a viable level at which to establish success criteria.

The involvement of both project management and operations management in the achievement of “project success”

Exhibit 1: The involvement of both project management and operations management
in the achievement of “project success”

It is not being suggested here that project success is somehow a “better” level at which to establish success criteria. Both project success and project management success are important to any project. If a project achieves project success without project management success, there is the inevitable conclusion that even greater benefits could have been realized. On the other hand, if project management success is achieved without project success, then the owner or sponsor has failed to obtain the benefits that the project was designed to provide. And that brings us to the third level of success.

Level 3. Consistent project success – Were the right projects done right, time after time?

As the focus moves from project management success, through project success to consistent project success, a completely new set of criteria come into play, as adjudged by different groups of stakeholders.

At this level, a discussion of success criteria is one that embraces the whole organization, and that is inevitably influenced by its chosen market and strategic intent. For operations-driven organizations (such as financial services companies, or mass manufacturers) consistent project success, in such areas as effective overall IT expenditure and new product development, can lead to competitive advantage. For project-based organizations (such as engineering contractors, defense suppliers or turnkey IT systems providers), consistent project success can lead to profitable expansion.

In recent years there has been a growing interest in project portfolio management for new product development (e.g. Cooper, et al, 2001), specifically for R&D (e.g. Matheson & Matheson, 1998) and generally for project spend in organizations (e.g. Artto, et al, 2001). But many organizations, particularly in traditional project-based industries, do not adopt a portfolio approach. Thomas and Jugdev (2002) have emphasized that long-range competitive advantage is enjoyed by those organizations that make the best use of their strategic assets (i.e. resources) and all organizations, regardless of industry must necessarily be concerned about the effective and efficient use of their scarce resources.

Project metrics: their role and some commonly encountered problems.

One practical implication of this discussion of three different levels of success criteria is that an organization would do well to monitor its performance using a “suite” of project metrics that incorporates all three levels of success, if it is serious about understanding and improving its success in the field of projects. As Exhibit 2 shows, each of the different levels of success is of interest to different levels of the corporate hierarchy, and each is visible after different amounts of time have elapsed relative to the project duration. No significant studies have as yet been published about the nature and extent of project metrics that are in general use, although Atkinson (1999) and Lim and Mohamed (1999) each argue that the need for multi-layered project success criteria is intimately linked to the need for more comprehensive metrics. Unpublished research within the Human Systems Network, however, supports this view and further indicates that few organizations are happy with the metrics that they use. Two phases of this research will inform the conclusions reached in this paper.

Metrics lie at the heart of management information systems, and it is no different for projects. Accepted wisdom about measurement can be summed up in the phrase, “If you want it, measure it. If you can't measure it, forget it.” (Belasco & Stayer, 1993, p 216). The clear implication is that the primary function of information systems and metrics is to assist in control. The current fashion for “dashboards” makes this explicit. “While driving a car, it makes good sense to pay close attention to what you are doing, looking at the road ahead and checking who is on either side of you and who is visible in the rearview mirror. A good driver will glance at the dashboard of the car as they proceed. This dashboard holds the essential information needed to drive the car, the speedometer, the oil pressure gauge, the gas gauge, and a few others.” (Eckes, 2001, p 21).

Measurement of project success. Organisational commitment and time elapsed

Exhibit 2: Measurement of project success. Organisational commitment and time elapsed.

The primary uses to which organizations put project management metrics are to establish a baseline for control purposes and to predict future costs (Egberding & Cooke-Davies, 2001). The least often used purposes are to assess corporate capability and to identify opportunities for improvement.

When looking at metrics in a little more detail, it is clear that organizations keep many different types: financials, time, quality, technical performance, resources used and available, project benefits, human factors and composites (dashboards, traffic lights etc.) (Egberding & Cooke-Davies, 2001). Indeed, there is much “good practice” around where metrics are concerned, although only one organization (a U.S.A.-based defense contractor) was able to lay claim to “best practice” in each of the four areas that will be described below.

Alongside these pockets of good practice, however, there is also considerable dissatisfaction with the metrics, especially from the financial community who tend to distrust the accuracy and objectivity of many of metrics produced from within the project management function. Composite metrics (such as “traffic light” indicators), while much liked by senior management, are also distrusted for their subjectivity and openness to bias. Where metrics are kept of “benefits management” they evoke almost universal dissatisfaction. There appears to me much room for improvement.

The three levels summarised

A detailed review of the literature on project success criteria and success factors allows the material that has been introduced so far to be summarised in Exhibit 3. The information has been derived from a detailed analysis of 44 published items of research into project management success, most particularly the excellent tabular summaries in Morris and Hough (1987), Belassi and Tukelv (1996) and Crawford (2001, Appendices C and D.

Characteristics of “best practice” in project management metrics.

Having considered the link between project success and organizational strategy, and reviewed the role that project management metrics play in project success, it is now possible to turn to the question that this paper set out to answer, “What are the characteristics of a suite of project management metrics that is capable of supporting an organization's efforts to improve project success?”

The characteristics are grouped together into four clusters. These do not represent four different sets of actions to be taken, but rather four different aspects of a well-rounded set of organizational “vital signs” that indicate the health of the project management efforts throughout the organization. They can be considered as “quality criteria” for an organization's metrics.

Three levels of project success summarised

Exhibit 3: Three levels of project success summarised.

Cluster I: “Completeness and Relevance”. A comprehensive hierarchy of metrics that provides suitable information for the right people to make the right decisions.

1     People at different organizational levels and in different job functions each require different sets of metrics to inform their decision-making. Different organizational levels are concerned with different kinds of decisions, and so need different levels of “roll-up” At a minimum, three levels can be distinguished: (a) business unit or portfolio, (b) programme or project sponsor, and (c) project manager and the project team.

2     The assessment of multiple projects is qualitatively different from the assessment of a single project many times over, and so additional analysis is required in addition to any “roll-up” involved in #1. At the top level, the metrics should cover more ground than simply the sum of the metrics from individual projects and programs within the portfolio. Metrics such as productivity, attrition, and process maturity are also relevant at this level.

3     Different job functions are interested in different aspects of project success. In addition to the different levels advocated in #1, a number of different job functions focus on different aspects. An effective suite of metrics provides each job function with those metrics that allow them to do their job effectively.

4

Cluster II: “Reliability of Data”. An infrastructure of processes and systems that collects necessary and useful metrics as efficiently as possible, and that is continually improved.

5     Metrics are only as reliable as the processes that produce them. Thus, if the process is distrusted, so will the metrics be. To give two examples, “traffic light” reporting (red, amber, green status) is often liked by senior management, because it allows the compact visual presentation of extensive amounts of data in a way that is easily absorbed, but it is distrusted elsewhere because of the variability of the judgements that influence the chosen colour. Secondly, the quality of time and resource forecasts is highly dependent on the quality of the planning process, and the availability of accurate data from past projects on which estimates can be based.

6     Multiple data flows lead to additional work reconciling information used as the basis of different metrics. This can be either hindered or helped by ERP systems. The accuracy of project financial data appears to be improved by the introduction of a well-planned ERP system, but individual project teams need both careful training and ongoing support if the data available to projects for financial decisions is to match the rolled-up data being provided through financial management.

7     Best Practice” separates the capture of data (done once) from the production of reports (done as automatically as possible). This calls for established disciplines, agreed definitions and well-thought-out IT/IS. Ideally, once a keystroke has been captured, it does not need to be re-entered into any system.

8     Most organizations have considerable scope for reducing cost and effort in gathering and reporting metrics.

The search for greater efficiency, accuracy and immediacy forms a continuous activity in the best run project-based businesses.

Cluster III: “Effective Use”. A culture that encourages metrics to be used to improve the quality of decision-making in the organisation.

Metrics are of value only to the extent that they encourage people to make high-quality decisions about projects, so as to improve the specific aspects of project success for which they have responsibility and authority. This cluster of characteristics is, in many ways, the most important, since it indicates the health of the human heart of an organization.

Thomas and Tjåder (2000) have pointed out that “control” and “learning” assume two very different paradigms. As we have already seen, control is a feature of mechanical systems that are actively steered by an external “driver”. Learning, on the other hand is a feature of higher organisms and is, moreover, an activity that is of necessity self-directed from within the organism. These two paradigms coexist uneasily together, and Melgrati and Damiani (2002) have demonstrated that it is the “control” agenda that dominates. Organizations are even less satisfied with their ability to transfer lessons learned from one project to others than they are with their suites of project management metrics!

9     Communications lies at the heart of complex organizational responses, and metrics play an important role in shaping conversations. This is the primary means of reconciling the conflicting agendas of control and learning. Decisions are made in conversation, and a well-crafted suite of metrics promotes dialogue that both encourages the right decisions to improve success, and also that results in the individuals involved in the dialogue learning something of benefit to the organisation.

10    Studies have shown that trust matters to project success (Construction Industry Institute, 1993) and conversations triggered by metrics can either foster or hinder the creation of trust. Thus metrics should be valued by and visible to all those who contribute to them.

11    The flow of metrics around an organization, coupled with different levels of confidence in the metrics, means that decision-making can be opaque and project management's input undervalued. This can be especially true in two situations: financial data emanating from the project management function that is at variance with financial data emanating from the financial function, and estimates (particularly of either time or benefits) that are repeatedly shown to be optimistic, and can thus continually be challenged.

12    If the control function of metrics is overemphasised at the expense of learning, and coupled with a tendency to “micromanagement” on the part of governance bodies, there is likely to be a tendency to distort the metrics that are provided upwards.

Cluster IV: “Continual Improvement”. Explicit improvement targets for project success that relate to organisational objectives.

13    Improvement targets are set for each of the three levels: project management success, project success and consistent project success. These targets should be explicitly related to current annual business goals so that the benefits of improvement are directly linked to organisational strategy.

14    Ownership is clearly established for each of the targets at each level, distinguishing clearly between governance (sponsorship etc.) and management of the project. (Turner &Keegan, 1999).

15    Decision-making follows the principle of “subsidiarity”, with decisions being taken at the lowest level at which the decision-maker is in possession of all sufficient and necessary information. Aside from published assertions about the importance of “authority” and “empowerment”, evidence for this is provided by proprietary quantitative evidence showing the correlation between team empowerment and cycle times in pharmaceutical research and development and by research into the maturity of project management in different industries (Cooke-Davies & Arzymanow, 2002).

Concluding comments: the role of metrics in developing project management capability.

An assessment of the suite of project management metrics that an organisation uses from the point of view of the four clusters of characteristics described in this paper performs two functions at the same time. Firstly, it provides a window into the quality of the measurement system that is used by different people to inform the decisions that they make about projects, and thus to point to areas that can be improved. Secondly, and perhaps more importantly, these four clusters of characteristics highlight areas of strength and weakness in an organisation's overall project management capability by indicating the extent to which vital signs can be monitored and changes responded to.

For both of these reasons every organisation that wishes to improve its execution of corporate strategy would do well to devote some energy to assessing its suite of project management metrics.

References

Anbari, F. T. (2003). Strategic implementation of six sigma and project management. PMI Global Congress 2003 -Europe. Philadelphia, PA: Project Management Institute.

Artto, K. A., Martinsuo, M., and Aalto, T. (2001) Project Portfolio Management. Strategic management through projects. Helsinki, Finland: Project Management Association Finland.

Atkinson, R. (1999). Project management: cost, time and quality, two best guesses and a phenomenon, its time to accept other success criteria. International Journal of Project Management 17(6) 337-42.

Belasco, J. A. & Stayer, R. C. (1993) Flight of the Buffalo New York: Warner.

Belassi, W., & Tukel, O. I.. (1996). A new framework for determining critical success/failure factors in projects. International Journal of Project Management 14 (3) 141-51.

Construction Industry Institute. (1993). Cost-trust relationship. U.S.A.: Construction Industry Institute.

Cooke-Davies, T. J. (2001) Towards improved project management practice: Uncovering the evidence for effective practices through empirical research. USA: Dissertation.com.

———. (2002). The “Real” Success Factors on Projects. International Journal of Project Management 20(3) 185-90.

Cooke-Davies, T. J., & Arzymanow A. (2002). The Maturity of Project Management in Different Industries. Proceedings of IRNOP V. Fifth International Conference of the International Network of Organising by Projects. Rotterdam: Erasmus University.

Cooper, R. G., Edgett, S. J., & Kleinschmidt, E. J. (2001) Portfolio Management for New Products. 2nd Edition ed. Cambridge, MA: Perseus.

Crawford, L. (2001). “Project Management Competence: The value of standards.” Henley Management College.

de Geus, A. (1997) The living company: habits for survival in a turbulent business environment. Boston, Mass: Harvard Business School Press.

De Wit, A. (1988). Measurement of project success. International Journal of Project Management 6(3) 164-70.

Eckes, G. (2001) The Six Sigma Revolution. How General Electric and others turned process into profits. New York, NY: John Wiley & Sons.

Egberding, M. & Cooke-Davies, T. J. (2002) GTN Metrics Survey: preliminary report on Findings Unpublished Work

Egberding, M. & Cooke-Davies, T. J. (2001) A quick summary of metrics pilot survey and other relevant background Unpublished Work

Frankl, V. E. (1959) Man's search for meaning. New York: Washington Square Press.

KPMG. (1997). Profit-Focused Software Package Implementation, KPMG, London.

Lim, C. S., and M. Zain Mohamed. (1999). Criteria of project success: an exploratory re-examination. International Journal of Project Management 17.(4) 243-48.

Matheson, D. & Matheson, J. (1998) The smart organisation. Creating value through strategic R&D Boston: Harvard Business School Press.

Melgrati, A., and Damiani, M. (2002). Rethinking the project management framework: new epistemology, new insights. Proceedings of PMI Research Conference. Philadelphia, PA: Project Management Institute.

Morris, P. W. G. (1994) The management of projects London: Thomas Telford.

Morris, P. W. G. & Hough, G. H. (1987) The Anatomy of Major Projects. A study of the reality of project management London: Wiley.

O'Connor, M. M, and Reinsborough, L. (1992). Quality projects in the 1990s: a review of past projects and future trends. International Journal of Project Management 10(2) 107-14.

Project Management Institute. (2000). A Guide to the Project Management Body of Knowledge. 2000 ed ed. Philadelphia: Project Management Institute.

Schwaninger, M. (1997) Status and Tendencies of Management Research: a Systems Oriented Perspectives. In Multimethodology. The theory and practice of combining management science methodologies. Chichester: John Wiley and Sons.

Thomas, J., & Jugdev, K. (2002). Project Management Maturity Models: The Silver Bullets of Competitive Advantage? Project Management Journal 33, (4) 4-14.

Thomas, J, & Tjåder J. (2000). On Learning and Control - Competing Paradigms or Co-existing Requirements for Managing Projects in Ambiguous Situations? IRNOP IV. Sydney: University of Technology in Sydney.

Turner, J. R, & Cochrane R. A. (1993). Goals-and-methods matrix: coping with projects with ill defined goals and/or methods of achieving them. International Journal of Project Management 11(2) 93-102.

Turner, J. R., & Keegan, A. (1999). The Versatile Project-based Organisation: Governance and Operational Control. European Management Journal 17(3) 296-309.

This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI or any listed author.

© 2004, T. J. Cooke-Davies
Originally published as a part of 2004 PMI Global Congress Proceedings – Prague

Advertisement

Advertisement

Related Content

Advertisement

Publishing or acceptance of an advertisement is neither a guarantee nor endorsement of the advertiser's product or service. View advertising policy.