Abstract
The question of whether risk management contributes to IT project success has occupied both academics and practitioners for a long time. Regarding the research on risk management, a distinction should be made between the evaluation approach that considers risk management to be an after-project analysis process with the aim of determining risk factors, and the management approach that considers risk management to be a management instrument to support decision making during a particular project. Within the context of this management approach to risk, this paper investigates both the claim of contribution to IT project success together with the validity of the assumptions on which it is based. Building on the conclusions of this investigation, a new model is proposed regarding how risk management contributes to project success. A validation of this model is a case study of an Enterprise Resource Planning (ERP) software implementation project indicating that intermediate factors of stakeholder communication and collaboration play an important role in the relation between risk management and project success.
Introduction
Risk management does not contribute to project success. Or perhaps it does? This question has occupied both academics and practitioners for a long time. Despite the extensive research conducted in this area, it remains difficult to establish the influence of actions that are meant to prevent something else from happening. Especially in the area of Information Technology (IT), where projects have a long history of failing, there is a high interest in risk management. So, what is it we actually know about the contribution of risk management to project success? Does the use of risk management lead to more successful projects?
This paper contributes to the live debate by presenting empirical evidence that either supports or opposes the claim that risk management contributes to IT project success. This paper starts with the discussion that within risk management a distinction is to be made between an evaluation approach and a management approach to risk management. Subsequently, the concept of project success in the context of IT projects is surveyed. Approaches to project success vary across publications, ranging from the traditional definition of project success (i.e., compliance to time, budget and requirements criteria), to broader, multistakeholder-oriented success definitions.
Following is a focus on the relationship between the management approach to risk management and its contribution to project success. Recent publications are analyzed to look for empirical evidence of the contribution of risk management to project success in combination with the assumptions on which risk management is based. The analysis leads us to summarize that although we have learned more about risks in IT projects and their impact on project success in recent years, there is little empirical evidence whether managing our project risks contributes to project success. In response, a model is presented that proposes a different view on how risk management influences project success. To conclude, the first results of empirical work will be reported, with an aim toward validating this model and providing insight into the mechanisms at work in the relation between risk management and project success.
Risk Management and Project Success
Two Approaches for Project Risk Management
The Evaluation Approach
For quite some time now, researchers have had a common interest in the area of risk and uncertainty in IT projects. The interest goes back as far as the late 1970s, with publications from, for example, Alter and Ginzberg (1978), Zmud (1980), and McFarlan (1981). These authors, as well as others who followed (e.g., Keil, Cule, Lyytinen, & Schmidt, 1998; Akkermans & van Helden, 2002; Han & Huang, 2007), consider risk management to be an ex-post evaluation process. The aim of such a process is to quantify the risks and find the causes of software project failure. This information is then used in the next project. Exhibit 1 presents this process graphically:
Known risk factors are the input for a project.
The project risk management process collects information about the risks and failure of the project, which leads to new risk factors.
These new factors are added to the list of known risk factors, together forming the input for the next project.
Exhibit 1: The evaluation approach to project risk management.
In this paper, this approach will be referred to as the evaluation approach. The evaluation approach assumes that if the risks and their causes are known, they can and will be managed, which is likely to lead to a positive effect on the project outcomes. The aim of the evaluation approach is to simulate project predictability by using the information regarding the risks and causes of project failure gathered in past projects in a new project. The assumption is that projects are comparable in the sense that information about risks can be generalized and used in future projects.
The Management Approach
The second approach formulates project risk management as a sequence of activities with the aim of gathering information about situations that may or may not occur (Boehm, 1991; Chapman & Ward, 1997; Pich, Loch, & DeMeyer, 2002). The sequence of activities that characterizes project risk management is identifying risks, analyzing risks, defining action, implementing action, and monitoring the situation. This sequence is described in detail by, for example, Del Caño and Pilar de la Cruz (2002) and in A Guide to the Project Management Body of Knowledge (PMBOK® Guide)—Third edition (Project Management Institute [PMI],2004). This sequence of activities is executed during the project with the intention to support and improve the management of the project by determining which actions should be taken. Exhibit 2 presents a graphical representation of this approach. In the context of projects and project success, the assumption is that better information leads to better estimates of the amount of time and money needed to complete the project. This better information provides a better insight into what should be delivered by the project (Chapman & Ward, 1997). By improving the project planning, budget and design, project risk management is expected to contribute to the success of the project.
Exhibit 2: The management approach to project risk management.
When is a Project a Success?
Traditionally, delivering on time, on budget, and according to requirements has been considered the criteria by which project success is measured. Despite the fact that this manner of measuring project success is currently subject to widespread criticism, these criteria are still often used in publications on project success in IT projects (Standish Group, 1999; Royal Academy of Engineering [RAEng], 2004). The criticism refers to three points, which are related to the assumptions on which this definition is based:
The amount of time, the budget, and the project's requirements can be set at the beginning of the project
The project's success is the same for each project stakeholder
The project's success can be determined at the moment the project has produced its deliverables
Turner and Cochrane (1993) state that the time-budget-requirements definition of project success focuses solely on the interest of the vendor or supplier, and not on the client. Some years earlier, De Wit (1988) had stressed the importance of including various stakeholders’ perspectives in defining project success. It is therefore remarkable that the traditional way of defining and determining project success is still very common in reports on project success and its relation to risk management (Standish Group, 1999; RAEng, 2004). Setting time and budget limits and defining requirements takes place at the beginning of the project, when uncertainty is at its highest (Pinto, 2007) and when it may be almost impossible to set realistic limits and goals, especially in IT projects (Turner & Cochrane, 1993). Project success may not be measured accurately by comparing limits of time and budget set at the beginning, with actual values of these limits at the end of the project. Therefore, this paper takes a view on project success in which project stakeholders define the success of the project following the approach started by De Wit (1988) and Turner and Cochrane (1993).
What Practice Tells Us About Risk Management
Project risk management (PRM) is a process with the intention to identify, analyzes, and manage events and situations that potentially form a threat to the success of the project. The PRM process consists of a limited number of activities. The typical activities of PRM are: risk planning, identification, analysis, and control (Del Caño & Pilar de la Cruz, 2002; PMI, 2004) (see Exhibit 3). It is assumed that the complete sequence of project risk management activities is used iteratively during the project. PRM is based on the rational decision-making assumption—that is, it assumes that all risks and uncertainties can be managed. These assumptions make PRM appear as an effective process with a positive impact on meeting project objectives and therefore on project success (Chapman & Ward, 1997).
Exhibit 3. Project risk management: the general approach.
Research has shown these assumptions are not always correct. Uncertainties, risks for which there is no classical or statistical probability distribution available (Holt, 2004), cannot be managed, at least not with the above-described PRM process (Pender, 2001; Pich et al., 2002). Furthermore, people in projects often fail to comply with other implicit assumptions of rational decision making. Kutsch and Hall (2005) have shown that, when faced with risk and uncertainty, project managers in IT projects may deny the presence of risk and uncertainty, avoid it, ignore it, or delay actions intended to respond to risk and uncertainty. Flyvbjerg et al. (2003) have shown that people in projects deliberately overestimate the benefits of the project, and at the same time they underestimate risks and uncertainties. When PRM activities are executed, various authors (Besner and Hobbs, 2006; Raz, Shenhar, & Dvir, 2002) find that not all PRM activities are executed, and that the prescribed sequence of PRM activities is not followed. Therefore, it can be concluded that only in a limited number of situations PRM will function as it is assumed to function.
The management approach to risk management has yet to lead to conclusive evidence about its contribution to project success. Based on publications over the last 10 years, it has been concluded that the empirical knowledge is still anecdotal and largely based on how risk management is assumed to work instead of on how it is actually used in project practice. Some authors report a positive impact of risk management activities on issues such as a timely project delivery, the estimations of the resources required to perform a task, and the number of task failures (Ropponen & Lyytinen, 1997; McGrew & Bilotta, 2000). Several other authors have referred to the characteristics and behavior of individual project stakeholders being of importance in relation to risk management and project success. Gemmer (1997) states that effective risk management requires functional behavior of the stakeholders, which means that they may not necessarily comply with the risk management process. Dey, Kinch, and Ogunlana (2007) state that, generally, stakeholders must be involved in the risk management process, whereas other authors are more specific about who needs to be involved, arguing that the involvement of users and top management in particular is crucial for the project's success (e.g., Jiang & Klein,1999). Jiang, Klein, and Means (2000) conclude that building consensus among stakeholders and stimulating communication with external stakeholders adds positively to team performance.
The management approach generally considers risk management as a process consisting of well-defined steps of identification, analysis, response, monitoring, and control. Less is known, however, about what occurs during the risk management process, which risk management practices are used within a project, which stakeholders are participating in these practices, how risk management practices influence stakeholders, and how risk practices influence project success. These are relevant questions, to which the risk management approach does not provide satisfactory answers, and neither does it give an accurate representation of how stakeholders actually behave.
How Risk Management Might Work in Practice
Building on earlier research in which it was concluded that often in projects the complete risk management process is not executed (Raz, Shenhar, & Dvir, 2002; Besner & Hobbs, 2006) and that project managers often do not decide rationally (Kutsch & Hall, 2005) when dealing with risk, an alternative model for the influence of risk management on project success is proposed here. Following literature mentioned previously in this paper (e.g., De Wit, 1988), it is assumed that project success is determined by each project stakeholder individually. This model is then validated in a case study, of which the first results are presented here.
An Alternative Model
The proposed model builds on literature by Besner and Hobbs (2006), and takes various project risk management practices as the starting point. Rational decision making is not assumed in this model; practices may or may not be used and they may be used in a random order. It is assumed that each project risk management practice may have an influence on project success. Building on the management approach to risk management (Del Caño & Pilar de la Cruz, 2002; PMI, 2004), the model investigates the following risk management practices: risk management planning, risk identification, risk registration, risk analysis, risk allocation, risk reporting, and risk control. Building on Turner and Cochrane (1993) and Agarwal and Rathod (2006), the proposed model takes project success to be a perception of an individual project stakeholder. The research investigates six indicators for project success: (1) delivery on time, (2) delivery within budget constraints, (3) delivery according to requirements, (4) stakeholder satisfaction, (5) project member work satisfaction, and (6) future potential of the project result for the organization. The relationship between risk management practices and project success is determined by intermediate factors. Building on literature by Rijsenbrij, Bauer, and Kowenhoven. (1993) and Chapman and Ward (1997), the model assumes and focuses on project stakeholder communication and project stakeholder collaboration as the important intermediate factors. Due to the exploratory character of the research, other intermediate factors that may affect project success are included in case they appear in the results. The model is presented in Exhibit 4.
Exhibit 4: Effects of project risk management practices on project success.
Results From a Case Study
The model has been validated in a case study in which interviews were held with stakeholders from an ERP software implementation project at a large international company in the food industry headquartered in the Netherlands. After a duration of a little over a year, this ERP project was delivered in the last quarter of 2008. The stakeholders interviewed were the project manager, a technical consultant, a project methodology consultant, and a senior user. The project was executed under the responsibility of the company. All risks for late delivery, budget overruns, and so on were carried by the company. IT suppliers were all working on time and on a material basis.
All interviewed stakeholders considered this project a success. There was little variation in their definition of success: stakeholder satisfaction and delivery according to requirements were among the top three for all stakeholders. On-time delivery was among the top three success factors only once, and the project was delivered on time, and close to budget limits. Statements by stakeholders that indicate the project success: “The organization resumed its original level of production just 1 week after the go-live of the new ERP system,” and “We did this in a little over 1 year, where normally this kind of project takes at least 2 years.” Scope changes occurred during the project, generally leading to the exclusion of functionality or the implementation of “short cuts,” which would have lowered the quality of the implemented solution. In terms of functionality provided, the situation in some cases got worse. One example is that in a certain situation there was an EDI (Electronic Data Interchange) solution, but in the new situation people have to use the fax again. Users say: “We have regressed 10 years.”
The methodology used on this project specifically aims at ERP implementations, and was developed based on earlier experiences from ERP implementations. Its approach is to make an ERP implementation as predictable as possible. A more predictable project leaves less room for risks; a firm and clear project plan (Loch. DeMeyer, & Pich, 2006) is in place. One of the aims of the methodology is to create alignment between people, and in the philosophy of the methodology, alignment lowers risk. The methodology does not provide clear directives or guidance on how to perform risk management, nor does it provide tools or instruments for measuring risk management.
The project manager deliberately deployed risk management by holding risk identification sessions, analyzing risks, and evaluating them throughout the implementation stage of the project. These activities were planned for in the planning documents for the implementation stage. The use of processes was not based on a specific methodology. Common tools were used—for example, brainstorming sessions for risk identification, spreadsheets for risk registration, and high-medium-low probability and impact for risk analysis. Only the management of the project and chairmen of workgroups within the project were actively involved. Other project members, users, or project board members were not actively involved in this process. All investigated risk management practices were used, although not all of them were individually detectable. In general, it was one integrated activity, in which risks were identified, analyzed, weighted, and allocated to an action owner, and their status regularly evaluated. Because of the top-down approach to risk management, users participating in the project were not aware of the use of risk management practices, but they experienced the effects.
All stakeholders, except the representative of the users, indicated that the following risk management practices had an influence on the project result: risk allocation (strong influence), risk management planning, risk identification, risk analysis (moderate influence), and risk reporting (moderate influence, but having more potential). On the question of how these practices had influenced the result of the project, intermediate factors were mentioned that linked to the risk management process in general and to individual practices. Overall, the risk management process has the potential to create an overview for all project members about the current position. and it directs common actions, making them more effective. Furthermore, the risk management process provides possibilities for reflection and for evaluation of project status at specific points in the project. The practice “risk identification” relates to the creation of a common view, of project members, towards project risks. Risk analysis relates to the creation of awareness with certain stakeholders about the situation, and the creation of a feeling of urgency to take specific actions to diminish risks. Risk allocation relates strongly to stimulating people to take action and to managing and controlling them and their actions. Risk reporting was found to stimulate people to take action, but this practice was underutilized and therefore less effective. As a result, risks remained the responsibility of only a small group of people, and only they took action.
Conclusion
Research conducted recently (De Bakker, Boonstra, & Wortmann, 2009) indicates that over the last 10 years, the evaluation approach to risk management has provided new and valuable insights into the risk factors that have an impact on IT project success. Both technical risk factors and organizational risk factors, such as senior management support and user participation, are highly influential. However, these indications are not enough to conclude that the evaluation approach contributes to project success, because knowledge of the risks alone is not sufficient to contribute to project success.
Research conducted on the management approach to risk management does not lead to conclusive evidence either. Based on recent publications it can be concluded that the empirical knowledge is still anecdotal and largely based on how risk management is assumed to work instead of how it is actually used in project practice. Considering the assumptions on which risk management is based, it is remarkable that literature does not come to the conclusion that risk management may not work as assumed. The literature ought to at least have recognized that risk management is not being conducted as it could be in order to be effective, according to its basic criteria.
These conclusions have led to the proposition of a new model on how the management approach to risk management influences project success. This model takes various project risk management practices as the starting point. Rational decision making is not assumed in this model. Furthermore, it is assumed that each project risk management practice may have an influence on project success. The proposed model incorporates project success as a perception of an individual project stakeholder and considers the relationship between risk management practices and project success to be determined by intermediate factors. The model assumes and focuses particularly, but not exclusively, on project stakeholder communication and project stakeholder collaboration as important intermediate factors. Results from this first case study indicate that the risk management process as a whole, as well as individual risk management practices, contribute to the project result. Risk management creates a shared view of the situation, it stimulates joint actions and makes actions more effective.
Acknowledgements
The author would like to express his gratitude to the people of the case organization, whose names unfortunately cannot be mentioned here, to Albert Boonstra and Sheilina Somani for reviewing a previous version of this paper, and to the PMI Risk Management SIG, for their continuous support of this research.
