A conceptual framework for application of performance measurement as an early warning system in projects
an analysis on the case of the London Ambulance Service project
This study presents an overview of the concept of early warning signs in projects and explains how a performance measurement system can be utilized as an early warning signal for avoiding failure. An analysis will be done on the published assessments of a project, the London Ambulance Service, which failed to fulfill its goals. This analysis has been performed in order to illustrate the feasible problems pertaining to a real case and its consequences. The rationale behind this selection is not to offer criticism relating to this specific project's performance but to learn constructively from it and move towards a better practice. A statement has been made that, with application of a performance measurement system in the project phase, chaos and perhaps total failure in the operational phase could have been prevented. Also, a conceptual performance measurement system has been proposed, using the main problems in the project phase as a reference for addressing the dimensions of performance to be measured, objects to be controlled, and the indicators. The overall aim of this paper is to increase understanding of the concept of early warning signs in projects and offer a possible approach, which can assist project managers in taking timely preventive actions in order to avoid undesired outcomes.
Keywords: performance measurement; project failure; early warning signs; conceptual framework; London Ambulance Service
Despite the improvement of project management practice in recent years, there is still a considerable rate of project failure. Many authors have looked into this subject and various responses to this problem have been developed. In their research, Sauser et al. (2009) found that the root cause of failure in projects is often managerial rather than technical. Some of the main managerial failure factors are mentioned to be inadequate project risk analysis and weak definition of scope (Yeo, 2002; Kutsch & Hall, 2005). Of course, there are many more factors that might result in project failure, but the question is how one might approach the issues that lead to project failure? Is it possible to prevent these situations or will these factors inevitably result in project deficiencies? The challenge, according to Sauser et al. (2009), is that while studying a project's success or failure it is important to realize that factors that work well in one specific situation may not work in another, or, in other words: “one size does not fit all.” Thus, one is required to choose the correct appropriate management style to help predict the success or failure of projects.
The ability to react to unforeseeable events and take proactive actions by detecting the early warning signs, in addition to early identification of problems, can assist in preventing potential causes of failure in projects. It is clear that the higher the risk of upcoming events, the more crucial it becomes to be able to predict and take actions accordingly in order to decrease the threat of failure. There is a need for more careful planning, close monitoring, and strict control of large, high-risk projects (Couillard, 1995). Identification of early warning signs and relating them to the appropriate project problems and their causes can contribute positively to the prevention of undesired consequences. Consequently, it is important to start applying this knowledge and actively start implementing actions aimed at lessening the threats, and the sooner the better (Nikander & Eloranta, 2001).
There are many different project assessment tools that can be utilized as early warning systems. Examples are project reviews, project health checks, benchmarking, post-project evaluations, and project audits (Klakegg et al., 2011). The research objective of this study is to describe the functionality of performance measurement systems in projects as early warnings with the aim of preventing undesired outcomes and to test whether the use of this principle is feasible in actual projects.
In order to achieve the research objective, a brief literature review has been done on the concept of performance measurement and how it can be used as an early warning system in projects. Although many authors have written about the concept of performance measurement (Kaplan, 1990; Maskell, 1991; Rolstadås, 1995; Folan & Browne, 2005; Jafari, 2007; Vittorio et al, 2008; Almahmoud et al., 2011), not much is said about its linkage to early warning signs in projects and how this tool can be utilized as an early warning system. The rationale for focusing on this tool is that it is recommended in the literature (Andersen & Fagerhaug, 2002; Nikander, 2002) as a useful prediction method.
Throughout this paper, we have mainly used the results of the work done by Andersen and Fagerhaug (2002) on designs and application of performance measurement systems and the performance measurement framework developed by Vittorio and Frattini (2009). This is followed by an analysis based on the work performed by Finkelstein and Dowell (1996) on the problems that drove the case of the London Ambulance Service to total failure. Benefiting from knowledge relating to the project conditions and the problems that landed the project in specific circumstances, a conceptual performance measurement framework is proposed to better illustrate possibilities for how this project could have been carried out differently.
Performance Measurement as an Early Warning System
Projects do not experience trouble overnight. Usually, they proceed from “green,” to “yellow,” to “red,” and during this process early warning signs can indicate if a project is on its way to failing or if urgent changes are needed (Kerzner, 2011). According to Kerzner (2011), the earlier the warning signs are discovered, the more opportunities for recovery exist. Successful identification of the early warning signs specifies if a project can achieve its objectives according to the original requirements with minor changes, if it can be repaired by implementing major changes, or if it should be terminated.
Andersen and Fagerhaug (2002) state that the earliest warnings originate from monitoring the basic knowledge development inside the organization and that these often emerge a year or two ahead of when the real problems make themselves apparent. Some of the typical early warning signs in projects, according to Kerzner (2011), include: different opinions on a project's purpose and objectives; unhappy stakeholders and steering committee members; delayed decisions, resulting in missed deadlines; unrealistic expectations; poor change control processes; and so forth. We believe that the reason why a performance measurement system can be used as an early warning is that it will aid project managers in taking preventive actions by providing a clear view towards oncoming matters rather than looking backwards. Having a proper performance measurement system in place can serve as a tool for detecting these signs in order to assist project managers in implementing the appropriate actions. According to Spitzer (2007), performance measurement creates the basis for effective management, thus offering a method of implementing strategies and policies in an organization, in addition to improving decision-making and problem solving. In addition, Andersen and Fagerhaug (2002) specifically highlight the usefulness of performance measurement as an early warning system. In the next section, the process of designing a performance measurement system will be described.
Design of a Performance Measurement System
The first important issue that should be considered by project managers when attempting to measure a project's performance is to design a system that fits the context in which it will be used. The design of this system should be clearly in alignment with the project's environment. The environment is, according to Vittorio et al. (2007), defined as: (1) critical objectives of the project, (2) organizational and managerial practices adopted for the project process, and (3) characteristics of the project's tasks that are going to be internally taken. An appropriate definition of the standards against which to measure performance is necessary to ensure that the measurement system provides useful indications capable of correcting the course of action. In fact, there is a need for a proper “benchmark” to be in place in order to use it as a reference (Vittorio et al., 2007).
According to Andersen and Fagerhaug (2002), there are two performance measurement system design approaches: A top-down cascading method and a bottom-up design process. The rationale for the top-down approach, which perhaps constitutes the most widespread approach, is that top management knows best which strategy to follow, which objectives to strive for, and what aspects of performance to measure. The bottom-up approach, on the other hand, is based on personal responsibility and is well suited for designing a performance measurement system that every member of the organization feels ownership of, can relate to, and generally view as useful. Considering the advantages and disadvantages of these approaches, the authors encourage a combined approach, where the top-down and bottom-up approaches meet somewhere in the middle. This ensures both that the organization's guiding star, its strategy, will define the boundaries for the performance measurement system and simultaneously encourage ownership and use of the system by allowing the users to define the details of the system themselves. The process of designing a performance measurement system includes (Andersen & Fagerhaug, 2002):
- Understanding and mapping business structures and processes
- Developing business performance priorities
- Understanding the current performance measurement system
- Developing performance indicators
- Decide how to collect required data
- Designing, reporting, and performance data presentation format
- Testing and adjusting the performance measurement system
- Implementing the performance measurement system
According to Andersen and Fagerhaug (2002), steps 1 and 2 are more complex than necessary, and step 3 can even be eliminated in small organizations. Steps 5 and 6 can be simplified and merged for projects in which a mass of performance data is not required. This is also applicable to steps 7 and 8. However, because the aim of this study is to design a performance measurement system for projects in general, which also include complex and high-risk projects, it would be appropriate to approach each and every step dependently when designing a performance measurement system. But it should also be taken into account that in large, complex project organizations such a process might not be able to capture the complex web of objectives, links between units, and so on. In these cases, it is recommended to apply the system design process to the independent tasks and units and then try to aggregate upward.
We have decided to use the model designed by Vittorio and Frattini (2009), which is a framework for designing a performance measurement system for new product development projects, as our framework. Also, the steps are clearly illustrated in a graphic model and are easy to understand. The suggested framework is illustrated in Figure 1. Each element of this framework, which will be described in this paper, matches each step of the design instructions suggested by Andersen and Fagerhaug (2002). The specific step that matches each element is also illustrated in Figure 1. The main aspects of this framework, as depicted below, are objectives, dimensions of performance, control objects, indicators, and the process of measurement. The objectives of measurement are to create a loop of never-ending improvement; the main objectives mentioned by Andersen and Fagerhaug (2002) are presented in the framework.
The choice of objectives and the structure of the system influence the dimensions along which measurement is undertaken. Several authors suggest that these dimensions can be brought back to “the balanced scorecard approach.” (Kaplan & Norton, 1996; Vittorio & Frattini, 2009) Consequently, performance should be measured taking into account the financial perspective, customer perspective, innovation and learning perspective, and the business process perspective. The control objects in this framework are a set of objects whose performance should be kept under control. The objects that should be kept under control during the project are project teams, the functional department under which the project team performs, the individuals, and the stakeholders. The indicators can be either qualitative or quantitative. For example, project time and cost can be measured by quantities, whereas the measure of quality of communication must be presented qualitatively. Of course, the choice of indicators to be employed should be consistent with the objective against which project performance is measured (Vittorio & Frattini, 2009).
Figure 1: Performance measurement system framework (adapted from Vittorio & Frattini, 2009)
The measurement frequency can be chosen to be regularly, for example on a monthly or weekly basis, or at specific milestones. This choice will depend on the performance measurement system objectives, the dimensions of performance, and the control objects. In some cases, a new project might prove similar to previous ones, for which a performance record is available. In this case, we believe that standards can be used as a reference in order to prevent the recurrence of previous risks and mistakes.
Having designed the main framework for measuring project performance, there is a need for clarifying the constitutive elements of the measurement system and describe which dimensions of performance should be monitored by the performance measurement system, which indicators should be used to monitor each performance dimension and identify which organization level are being monitored, and also the indicators associated with each of these. In order to accomplish this, an objective must first be identified and the above-mentioned steps will then be followed according to this specific objective.
In the next section, the case of the London Ambulance Service project, which turned out to be a failure, will be subjected to analysis. The reason why this published case study has been chosen is that this project, mentioned by Williams (2002), is an example of a highly complex project, reputed to having experienced a disastrous outcome due to lack of appropriate management. We would like to discuss this case in order to identify which measures could have been undertaken differently in order to prevent the project's failure. Note that, in addition to this, the information published on this case is clear, detailed, precise, and the causes of failure, management conditions, and other important aspects needed for analyzing a case are provided in the published literature pertaining to this specific case.
London Ambulance Service (LAS): A Case of Total Failure
This section includes a brief overview of the case of the London Ambulance Service based on studies by Finkelstein and Dowell (1996) and Dalcher (1999). The reason we have chosen to analyze this case is that this project, according to Williams (2002), was a highly complex, long-duration project and experienced a number of serious problems during the operation phase. The purpose of this study is to discuss how this project could have been managed differently and in a more effective way.
The London Ambulance Service is referred to as the largest ambulance service in the world, covering an area of over 600 square miles and a population of approximately 7 million people. The service was divided into two sections: one providing routine patient transport and the other an accident and emergency service. In 1990, the LAS decided to commission the development of a computerized dispatch system in order to improve its performance. Prior to this, the service used manual methods for controlling the dispatch of ambulances. The details of a call were noted on paper and sent to a central collection point. The communication with ambulances before dispatch was done by telephone and once the ambulance was dispatched, via radio.
A requirements specification was prepared and an invitation to tender for the development was issued in February 1991. Many companies responded, although many commented that the proposed implementation date of January 1992 made the time scale very tight. Nevertheless, a supplier who offered to meet the deadline was chosen. Note that this supplier didn't have enough experience to carry out this kind of system. Price was a major factor in selecting the winning bid.
During the development of the new system (project phase) many problems occurred, including:
1. The delivery of software was frequently late
2. There was no proper project management
3. None of the people involved in the project had experience with PRINCE, the project management methodology that had been adopted for the development
4. Software developers made ad-hoc changes to the software in order to achieve user satisfaction
5. The users were not adequately trained and were skeptical about the benefits of the system
Due to the above-mentioned problems, the implementation of the system was pushed back until late 1992, but it became evident that many problems were still unresolved.
The computerized dispatch system went live at 7:00 a.m. on 26 October 1992. At the time of start-up, when there were few calls to deal with, the system worked satisfactorily and the staff was capable of managing the tasks, but as the number of calls increased, it became clear that there were major problems riddling the system. The problems that showed up in the operational phase included:
1. The system failed to eliminate duplicate calls, so sometimes more than one ambulance attended a scene.
2. The tracking of ambulances did not work properly.
3. Ambulance crews were confused about how to apply their status reporting system. This resulted in the system failing to produce a true picture of the ambulance fleet's status, thus making inappropriate allocations of ambulances to incident sites.
4. The delays caused people at incident sites to call again and consequently increase the load on the system.
Eventually, it became clear that the system failed to handle the amount of calls and consequently lost control of its fleet. On the same day this fact was established, the service reverted to a semi-manual system, using only part of the original software (i.e., the one for taking incoming calls). This functionality also failed several weeks later due to a program error, and the system was then completely abandoned.
The failure of this system attracted a great amount of media attention and there were allegations that approximately 20 people had possibly died as a result of the delays of up to 3 hours in ambulances arriving at incident sites. Later inquiries, however, stated that there was no evidence of deaths occurring due to the ambulance delays. The following aspects were identified as the main causes of system failure:
1. Neither the system nor the user was properly prepared for the implementation of the system.
2. The system contained design flaws.
3. There was an unrealistic reliance on correct information about ambulance location.
4. The project timescale had been too short.
5. The software was incomplete and effectively untested.
6. The implementation approach was ‘high risk’ and inappropriate and unjustified assumptions were made during the specification process.
7. There was a lack of consultation with users and clients in the development process, with immediate consequences for their “ownership” of the resulting system.
8. The poor fit of the system with the organizational structure of the ambulance service
9. Poorly designed user interfaces
10. Lack of robustness and straightforward “bugs” and errors
In the next section, a performance measurement system will be proposed for this project, indicating that if this system had been applied as an early warning system, perhaps many of the problems, and consequently serious failure, could have been avoided.
What Could Have Been Done Differently?
In the case of the London Ambulance Service it is clear that some issues could have been handled differently, and certain early warning signs could have been detected in advance to prevent encountering serious problems in the end. Many studies have been done on the case of the London Ambulance Service and analyses carried out to determine the reasons why this project ended in such a disaster (Hougham, 1996; Beynon-Davis, 1999; Dalcher, 1999; Fitzgerald & Russo, 2005). We would like to analyze this case from a management point of view and discuss how the application of a performance measurement system in the project phase of the London Ambulance Service could have contributed to preventing chaos and perhaps total failure in the operational phase.
In his study, Kerzner (2011) introduces a list of typical early warning signs in projects and indicates that the sooner early warning signs are detected, the more opportunities exist for recovery and for improving the possibility for successful projects. According to the case descriptions in the previous section, the specific problems in the operational phase can be matched to one of the typical early warning signs introduced by Kerzner (2011) (Table 1).
We would like to suggest that a performance measurement system might have been used as an early warning system in this project. The suggested system follows the framework presented in Figure 1. Each element of the framework is adapted to the specific condition and aspect in the LAS project. In order to do this, the main problems that occurred in the project phase will be used as a reference for addressing the dimension of performance, the objects to be controlled, the indicators of measured elements, and the suitable process of measurement. The system's components are presented in Table 2.
Table 1: Problems in the project phase as early warning signs.
|Specific problem in the operational phase||Early warning sign|
|Software frequently late||Delayed decisions resulting in missed deadlines|
|Ad-hoc changes by software developers to achieve user satisfaction||Poor change control process|
|Users not adequately trained and skeptical about the benefits of the system||Different opinions on project's purpose and objectives|
|Short time scale||Unrealistic expectations|
|Software not tested||Technical failure|
|Lack of consultation with the users and clients||Failure in progress reporting, poor morale|
|Unjustified assumptions in the specification process||Unrealistic expectations|
In this model, the dimension of performance that should have been measured to prevent problems in the operational phase of the project is mentioned. Each problem is related to one of the following areas, which are the components of the balanced scorecard:
1. Financial issues such as operating income, return on investment, and economic added value
2. Learning and growth issues such as employee satisfaction, employee retention, skills, and so forth
3. Customer issues such as customer satisfaction and customer retention
4. Internal business issues such as cost, throughput, and quality
Table 2: Components of the suggested performance measurement system.
|Problems in project phase||Dimension of performance||Control objects||Indicators|
|Software frequently late||Service delivery on time (business process)||Project team||Percentage of project concluded on time (earned value)|
|No proper project management||Project management effectiveness (business process)||Project team||Value of schedule variance and cost variance|
|No experience with PRINCE (project management methodology adapted for development) for people involved||Capability of applying selected project management methodology (learning and growth)||Project team||Project performance during first month|
|Ad-hoc changes by software developers to achieve user satisfaction||Level of changes in the software (customer)||Project team||Percentage of customer satisfaction|
|Users not adequately trained and skeptical about the benefits of the system||Capability of clarifying goals of the project for users and level of training (learning and growth)||Individuals||Number of positive users in the beginning of the project|
|Short time scale||Level of proper planning (business process)||Functional department /project team||Percentage of project concluded on time according to the initial estimations|
|Software not tested||Quality of software performance (business process)||Project team||Percentage of “bugs” and errors|
|Lack of consultation with the users and clients||Quality of communication with customers (customer)||Customer/project team||Percentage of client satisfaction through the development stage|
|Unjustified assumptions in the specification process||Level of justification and reality of estimations (business process)||Functional department||Percentage of achievements according to the assumptions at each decision gate|
In the proposed model, the dimension of performance for each aspect is matched to the area in which it fits. For example, “lack of consultation with users and clients in the development process” was a problem that occurred due to lack of a measurement system for customer satisfaction during the process.
The control objects, the objects whose performance should be kept under control, are set for each issue. For example, “too short project time scale” could be improved by controlling the project team's performance, their improvement during a specific period of time, and the estimates they would have made to identify the right finishing time of the specific tasks they were responsible for. The frequencies of measurement, which can be on specific milestones or regularly throughout the project, are identified according to each aspect. For example, in the case of a “too short project time scale,” if the project teams’ performance had been measured regularly, comparing the planned work to be done with the actual work done (earned value method), the flaws could have been identified in advance.
Since the project team was not sufficiently experienced in developing this specific kind of system, there were no internal standards for use as a reference; however, any general software development standard that defines and establishes the routine process for software development can be used as a reference in order to prevent common risks and mistakes.
Having applied this model, the problems in the operational phase could have been overcome to some extent or even totally prevented. An example of the problems in the operational phase is the failure of the system to eliminate duplicate calls and, as a result, dispatch ambulances to a scene more than once. According to the suggested performance measurement model, if the percentage of “bugs” and errors were measured, the software problems could have been identified in advance and, as a result, the above-mentioned errors could have been prevented in the operational phase. Each of the aspects mentioned in Table 2, in case they were noticed and acted upon, could have contributed to the prevention of the major problems that led the project to total failure.
Also, the level of customer satisfaction could have been evaluated during the project in order to identify their expectations in advance, thus avoiding surprising and unwanted events.
Discussion and Conclusions
Projects do not go from “totally perfect” to “totally disastrous” overnight. There are always specific signs that indicate that there will be a problem ahead. The sooner these signs are detected by the management team, the more possibilities exist for taking preventive actions. Some of the project assessment tools that are used in projects as early warning systems have been mentioned in this paper. Performance measurement is one of the tools that can assist project managers in the early identification of early warning signs. The example of the London Ambulance Service was chosen to better illustrate the problem and the possible solutions to it. In this specific IT project, the problems were clearly identified by looking back at what had already happened. This provided a good knowledge base for indicating what should have been measured, consequently identifying the early warnings, and which measures could have been implemented in a different manner.
It should be mentioned that we are fully aware of the fact that it is easy to look back on history and state what could have been done differently. We believe that the real challenge is to identify the early warning signs while the project is running and to be able to detect them early enough to take the right actions. As mentioned earlier, one size does not fit all, and this is not a general framework that can be utilized in all cases. There is a need for more enhanced awareness of project conditions, thus enabling project management to choose the appropriate indicators and measurement dimensions.
We would like to also emphasize that because this framework has not been tested in this specific project, there is no guarantee that an application of this framework could have definitely prevented failure. We should take into account that, in some cases the early warning signs are in fact sensed, but nothing is done about them. Here is when we should consider the human side, which can very much influence the process of detecting early warning signs. According to Holmes (2001), the London Ambulance Service organization had the culture of “fear of failure,” according to which senior management was continually under pressure to succeed. This could cause avoidance of even observing the early warning signs, let alone taking proactive actions to respond to them. Klakegg et al. (2011) also discuss the problem pertaining to why project assessment methods are not capable of identifying early warning signs in projects. They mention and discuss three particular areas that can contribute to this problem, including: complexity, understanding of risk, and interpersonal affects. Some of these issues are identified in the analysis done by Holmes (2002) on the case of the London Ambulance Service. For example, the London Ambulance Service organization had a history of severely problematical industrial relations between management and the ambulance crew. As a result, consultation with the ambulance crew during system validation was avoided. We conclude that not only was a systematic approach not applied to detecting the early warning signs in this project, but there was also a culture of ignorance, which led the project to such failure. Further studies will be carried out on how to detect the early warning signs in real ongoing projects and the human and organizational factors that should be considered in order for an early warning system to be effective.
Another interesting point is to identify at what stage in the project life cycle it is possible to detect the early warning signs. In the previous section it was mentioned that the supplier chosen to handle the London Ambulance Service project didn't have enough experience to carry out this kind of system and that the price was a major factor in selecting the winning bid. Looking at the bigger picture, it is somehow possible to see these two issues as early warnings that the allocated tasks will not be carried out with the prerequisite quality. We would like to conduct further research on the specific stage in the project in which the process of early warning detection should begin in order to prevent a series of problems in the following stages.
In this paper we hope we demonstrated a basic description of the concept of early warning signs in projects and how performance measurement can be utilized as one of the tools which, if applied as part of the management system, can help project managers to not only detect the early warning signs of oncoming problems, but also enable them to take preventive actions on time in order to avoid adverse outcomes.
Almahmoud, E.S., Doloi, H.K., & Panuwatwanich, K. (2011). Linking project health to project performance indicators: Multiple case studies of construction projects in Saudi Arabia. International Journal of Project Management, doi: 10.1016/j.ijproman.2011.07.001
Andersen, B., & Fagerhaug, T., (2002). Performance measurement explained: Implementing your state-of-the-art system. Milwaukee, WI: ASQ Quality Press.
Beynon-Davis, P. (1999). Human error and information systems failure: The case of the London Ambulance Service Computer-Aided Dispatch System Project. Interacting with Computers, 11, 699–720.
Couillard, J. (1995). The role of project risk in determining project management approach. Project Management Journal, (26)4, 3–16.
Dalcher, D. (1999). Disaster in London: The LAS case study. Engineering of Computer-Based Systems, Proceedings. ECBS ‘99. IEEE Conference and Workshop on 03/07/1999-03-12/1999, 41–52. Nashville.
Finkelstein, A., & Dowell, J. (1996). A comedy of errors: London Ambulance Service case study, Proceedings of the 8th International Workshop on Software Specification and Design, March 22–23, Schloss Velen, Germany.
Fitzgerald, G., & Russo, N.L. (2005). The turnaround of the London Ambulance Service Computer-Aided Dispatch system (LASCAD). European Journal of Information Systems, 14, 244–257.
Folan, P., & Browne, J. (2005). A review of performance measurement: Towards performance management, Computers in Industry, 56, 663–680.
Holmes, A. (2001). Failsafe IS project delivery. USA: Gower Publishing Company.
Hougham, M. (1996). London Ambulance Service, computer aided dispatch system. International Journal of Project Management, (14)2, 103–110.
Jafari, A. (2007). Project and program diagnostics: A systemic approach. International Journal of Project Management, 25, 781–790.
Kaplan, R. S. (Ed) (1990). Measures for manufacturing excellence. Boston: Harvard Business School Press.
Kaplan, R. S., & Norton, D. P. (1996). The balanced scorecard. Boston: Harvard Business School Press.
Kerzner H. (2011). Project management metrics, KPIs, and dashboards: A guide to measuring and monitoring project performance. Canada: John Wiley & Sons, Inc.
Klakegg, O.J, Williams, T., Walker, D., Andersen, B., & Magnussen, O.M. (2010). Early warning signs in complex projects, USA, Project Management Institute Inc.
Kutsch, E., & Hall, M. (2005). Intervening conditions on the management of project risk: Dealing with
uncertainty in information technology projects. International Journal of Project Management, 23, 591–599.
Maskell, B. H. (1991). Performance measurement for world class manufacturing. Cambridge, MA: Productivity Press.
Nikander, I.O. (2002). Early warnings: A phenomenon in project management, Dissertation of Doctor of Science in Technology, Helsinki University of Technology, Espoo, Finland.
Nikander, I.O., & Eloranta, E (2001). Project management by early warnings. International Journal of Project Management, 19, 385–399.
Rolstadås, A. (ed.) (1995). Performance management: A business process benchmarking approach. London: Chapman & Hall.
Sauser, B.J., Reilly, R.R., & Shehnar, A.J. (2009). Why projects fail? How contingency theory can provide new insights: A comparative analysis of NASA's Mars Climate Orbiter loss. International Journal of Project Management, 27, 665–679.
Spitzer, D.R. (2007). Transforming performance measurement: Rethinking the way we measure and drive organizational success. New York: AMACOM.
Vittorio, C., & Frattini, F. (2009). Evaluation and performance measurement of research and development: techniques and perspectives for multi-level analysis. United Kingdom: Edward Elgar Publishing Inc.
Vittorio, C., Frattini, F., Lazzarotti, V., & Manzini, R. (2007). Measuring performance in new product development projects: A case study in the aerospace industry. International Journal of Project Management, 38, 45–59
Vittorio, C., Frattini, F., Lazzaroti, V., & Manzini, R. (2008). Designing a performance measurement system for the research activities: A reference framework and an empirical study. Journal of Engineering and Technology Management, 25, 213–226.
Williams, T. (2002). Modeling complex projects. United Kingdom: John Wiley and Sons, Ltd.
Yeo, K.T. (2002). Critical failure factors in information system projects. International Journal of Project Management, 20, 241–246.
©2012 Project Management Institute