Project risk and uncertainty

Abstract

Evidence from a longitudinal case study and related research is used to show how methods drawn from cognitive psychology can help managers to identify the risks that may impact on projects at the strategic investment decision stage. The paper argues that such methods can be used to enhance the risk management of projects.

Introduction

This paper examines from a psychological perspective the challenges that managers face in assessing the likely risks and benefits to be derived from strategic investment opportunities. Evidence from a longitudinal case study and related research is used to show how methods drawn from cognitive psychology can help managers to identify the risks that may impact projects at the strategic investment decision stage. The paper argues that such methods can be used to enhance the risk management of projects.

First, strategic investment decisions are defined and areas of relevant literature are identified before focusing on the psychological dimension. Research on risk and the approach commonly used in project management for risk assessment is then presented.

The second section of the paper introduces qualitative methods drawn from cognitive psychology. This section also sets out the research approach taken to provide evidence of how such methods may be applied and to support practical conclusions.

The third section sets out the findings and limitations of this research. Qualitative methods are illustrated and evaluated and a project typology is suggested to facilitate risk identification. The issue of controllability is analyzed to explore how the learning from early stage risk assessment (pre-decision) can benefit project management post-decision, with practical implications for knowledge transfer from decision making to project management.

Literature Review

In early accounting literature, rooted in economics, the term capital budgeting was defined as “the allocation of scarce resources between alternative uses so as to best obtain objectives…over time” (Bromwich, 1976). This covered decisions on the total amount of capital expenditure a firm should undertake and the financing of projects, as well as the decisions about which specific investment projects to accept. The main methods of investment appraisal recommended, internal rate of return and net present value, were based on discounted cash flow (DCF) techniques.

The term capital budgeting also implied that capital expenditure decisions might be routinized into the overall budgeting process, usually within an annual planning cycle. In the case of investment in assets such as manufacturing equipment in established firms, where life cycles and capacity requirements may have been relatively predictable, this assumption may have been reasonable. However, with the more rapid change and complexity involved in advanced technology and the emergence of new knowledge-based industries, a fundamental change in our thinking about investment in projects has become necessary.

A view that DCF techniques misplaced the emphasis of capital budgeting was expressed (King, 1975) and a broader view advocated with multiple steps in the decision making process, from triggering, screening, and definition to evaluation, transmission, and decision. However, it took time for this view to become widely accepted in the accounting and finance literature.

The need to focus business planning more externally on the competitive environment and the shortening of time available to identify and evaluate new opportunities is well documented in the strategy literature, where investment decisions are more about the formulation and implementation of strategy. Recent research is concerned with strategic alignment of projects (Langfield-Smith, 2005).

Strategic investment decisions are still concerned with choosing “between alternative(s)…so as to best obtain objectives” (Bromwich, 1976), but involve a far broader consideration than the economics of the prospective project. Research in the 1980s and 1990s “focused on the fit between the use of DCF techniques for capital expenditure evaluation and specific contingencies of business strategy, external environment, information systems characteristics, reward systems structure, and degree of decentralization” (Langfield-Smith, 2005). The potential involvement of more managers in the process brings organizational behaviour, particularly psychology, into consideration.

Simon (1945) was one of the earliest and most acclaimed researchers to write about the psychology of organizational decisions. He identified the problems of assuming economic rationality in a process reliant on human behavior, and of the roles of authority and communication. Later editions of the work cited added sections on information processing, organizational design, and the selective perception of executives (Simon, 1976). However, many lessons from this work seem to have been neglected. This paper aims to put such issues firmly back on the agenda, as it is argued that they are even more relevant in the context of project management and accountability in today's corporate world.

Kahneman and Tversky (1979) developed prospect theory, which also contradicted the economic rational model of decision making. They found that people do not use probabilistic calculations to ascertain the economically optimal solutions, but that they exaggerate downside risks if they are risk averse and may exaggerate the chances of success when faced with upside risks, for example by buying a lottery ticket at low cost with little chance of a potentially high gain. This kind of behavior is just one of the “heuristics” found in the psychology of decision making. Tversky and Kahneman (1981) also found that people react differently to the same prospect if it is presented differently, known as the framing effect. Framing experiments focus on how the gains and losses of an option are presented from a zero or other reference point to give the decision maker a different impression of the same set of cash flows.

This work on the psychology of intuitive judgment has been extended to include many examples of heuristics and types of bias in decision making (Gilovich, Griffin, & Kanhneman, 2002). One of the newer examples is the affect heuristic, which deals with the emotive or intuitive response to certain words or phrases used in describing or framing a prospect (Slovic, Finucane, Peters, & MacGregor, 2002). It extends the framing concept to include the language used to define a project in addition to the presentation of financial data. When decision makers assess the risks of a project opportunity they may respond differently to the same prospect if colorful or emotive language is used.

It may be recognized that such language is used to advantage in marketing consumer products, but this affect heuristic could also be used by project champions to elicit a positive response when trying to sell their ideas to executives to gain approval for projects or by others in the organization wishing to create a negative response for example to avoid organizational change.

Bazerman (2006) has worked on the area of managerial judgment in decision making from an organizational behavior perspective for over 25 years, several times adding to and editing chapters from the first edition published in 1985. The focus of the work is on helping individuals overcome personal bias in order to make better managerial decisions Hastie and Dawes (2001) also examined the role of personal bias in decision making and the conflict between mathematically based decision methods using probability theory and human information processing. They examined the psychology of judgment in investment decisions, but they tended to focus on individuals making stock market investments rather than business managers evaluating strategic project opportunities.

While much of this work has been updated to place more emphasis on investment decisions, the problem is that it is still largely focused on the individual, neglecting the strategic context of groups of managers acting together in decision making processes in large organizations. The affect heuristic was found to influence managers in the mining industry in their perceptions of the risks and benefits of an activity (Dunbar, 2007). The focus on the individual was acknowledged by Dunbar, who suggested that future research on framing effects might explore how this affects decision making within groups. Simon (1945) recognized the role that organizational structure and culture have in creating the decision making environment for managers.

Bower (1986, first published in 1972) first examined the group process, which he named the “impetus” stage in decision making, to describe the rate of progress of the proposal upward through the organizational hierarchy. Mintzberg, Raisinghani, and Theoret (1976) also identified the importance of internal politics and persuasion to gain consensus. Neither study examined the psychological phenomena of framing or heuristics explicitly in their research, coming from a more sociological perspective.

Helliar, Lonie, Power, and Sinclair (2001) tested the framing effect by designing alternative scenarios in a questionnaire used to explore UK managers' attitudes to risk in decision situations. They found that “the framing of a decision in terms of gains or losses was critical in determining whether a manager would adopt a risk-seeking or risk-avoiding stance” (Helliar, Lonie, Power, & Sinclair, 2001, p. 87). They grouped the risks perceived by the managers they interviewed into four categories, financial, labor-related, strategic and miscellaneous. The strategic risks “ranged from concern about market share to fear of being taken over” (Helliar, Lonie, Power, & Sinclair, 2001, p. 85). This type of risk might be expected to be more emotive and therefore more subject to the affect heuristic than say financial risks, where variability of returns or trade credit risks could be quantified, though this was not specifically tested.

Collier, Berry, and Burke (2007) used exploratory cases and a survey to investigate risk management practices in UK companies. While there was no particular focus on strategic investment decisions or project risk assessment, there were interesting findings on managers' attitudes to risk. They found that “heuristic methods of risk management were used much more than the systems-based approach that is associated with risk management in much of the literature, at least at the corporate level” and “the methods in highest use were the more subjective ones (particularly experience)…reinforcing the role of the human actor over analytical techniques” (Collier, Berry, & Burke, 2007, p. 117). They also found organizational stance towards risk important in determining risk management practices.

The systems-based approach in the form of a nine-phase risk management process (RMP) is prevalent in the project management literature, as one might expect from an area that has its roots in engineering and management science (Chapman & Ward, 2000). The key technique for assessing the risks associated with a project is to estimate the probability of a specific source of risk affecting the project and assess the impact such effect might have. Chapman and Ward (2003) devoted a whole chapter to estimation as probability theory is not widely understood by business managers. The probabilities may be represented in a probability impact grid (PIG), which seeks to quantify the effect of risks on project outcomes (see example, Webb, 2003, p. 124). This formed the basis of early risk management software such as RISKMAN (Carter, Hancock, Morin, & Robins, 1994) and was expressed as:

Risk exposure = impact value x probability of occurrence

The model is simple, but the data inputs to the model rely on subjective judgment. It has been shown that when people are asked to estimate the frequency or chance of a risk or hazard occurring (perceived risk), they tend to overestimate low probabilities (compared to technical estimates based on known frequencies) and underestimate high probabilities (Slovic, 2000, p. 116). This can also apply to the impact or consequences, where people tend to overestimate the impact of high consequence events. When combined in a low-probability, high-consequence event, the risk exposure will be magnified. Recognizing the psychological limitations of subjective estimation and the instability of estimation over time (as people react to new information) is important in risk assessment and project management. The danger of turning the risk exposure into a number by multiplying two estimates together is that it gives the analysis a more scientific appearance than is really the case, implying a level of accuracy that may be misleading.

The other limitation in the application of this type of risk assessment is the completeness of analysis, as not all relevant sources of risk may have been identified. When projects fail it can be due to a risk that no one perceived when the project was evaluated. For this reason there are a number of approaches that might be taken at the risk identification stage to ensure that all significant risk sources are captured, including checklists. These may be generic (for example, Webb, 2003, p. 28) or specific to a type of project, such as new product development (Webb, 2003, p. 42). Other approaches include brainstorming, interviews, questionnaires, and so forth.

Risk assessment may be quantitative, using a risk model or simulation to estimate exposure, as described above, or it may be qualitative, “describing characteristics of each risk in sufficient detail to allow them to be understood” (Hillson & Murray-Webster, 2005, p. 20). Qualitative methods have the advantage of surfacing issues of risk attitude and sharing of ideas between participants, helping to develop a common language for risk description in a business team or organization. Best practice risk identification can also harness the emotional literacy/intelligence of participants (Hillson & Murray-Webster, 2005, p. 112).

The next section of this paper discusses three qualitative methods that can be used by academics in researching people's perceptions, and by practitioners and consultants as a method of risk identification, or both, in the co-creation of project relevant knowledge (action research). This section has been kept purposefully short and relatively jargon-free, so as to focus on the practical application of two of these techniques in the longitudinal case. The academic references provided may be followed up for a more detailed discussion of methods.

Methodology

This section of the paper sets out the qualitative methods that can be used both as research methods to elicit the personal constructs underlying how a person makes sense of a topic, and as a means for managers to identify sources of risk in relation to strategic investment opportunities. They are drawn from cognitive psychology and are all based on a theory of cognition. Personal construct theory (PCT) developed by Kelly (1955) is a theory of cognition that suggests that people make sense of their world by sorting each new experience into a set of bipolar constructs (opposite concepts) by reference to previous experience.

To assess the attractiveness of a new project opportunity an organizational member will recall features of previous projects, both approved and rejected, both successes and failures. In order to differentiate one project from another, sufficient information is needed to compare and contrast prospects on a number of features or characteristics. To assess the riskiness of the alternatives, the projects may be compared using a series of constructs that contribute to the overall level of risk. These may be regarded as project risk attributes or risk drivers. One such construct might be the level of novelty, such that a new project which is very similar to previous ones might be regarded as low novelty and therefore low risk. In contrast, a research and development project involving technical innovation and a new target market might have a high level of novelty and therefore a high risk attached.

Three methods of eliciting personal constructs that may be used separately or combined are the nominal group technique (NGT), a form of brainstorming involving both individuals and groups, repertory grid technique and cognitive mapping. For these methods to be applied in an attempt to identify group cognition or a shared understanding of a topic, in this case project risk, there should be a preliminary test to establish if Kelly's commonality and sociality corollaries apply (Kelly, 1955).

In the context of managers in an organization, this means ascertaining if they have sufficient common ground to reach shared views of project opportunities. Commonality means the individual managers will have shared or common understandings, thinking about the risk drivers in a similar way. Sociality means that while the members of the group may not always agree or hold a shared view, they can understand and appreciate the views expressed by others in the group and find ways to reach agreement even where there is not absolute consensus.

Where organizational members have worked together in the same organization for some time, such that they have shared experiences of previous projects evaluated and undertaken, these corollaries are likely to hold when they discuss new projects, even if their attitudes to risk differ. Working together in an established management team and having experience gained in the same industrial sector will auger well for reaching a shared understanding of project risk.

Ask the same people to give their views on topics such as politics or religion, where non-work experiences will have shaped their thinking, and one might expect considerable variation of views and difficulty reaching anything close to consensus. For this reason, these research methods are less appropriate for use with newly formed groups or with surrogates such as student groups. The researcher will need to assess the group profile and observe the team dynamics in order to ensure that the commonality and sociality assumptions are reasonable.

The first of the three methods, the nominal group technique (NGT) relies less on Kelly's underlying PCT principles, but where the aim is to elicit a group view, then commonality and sociality will help the process. NGT is a two-stage process, where a facilitator sets out the topic or focus of discussion and first tasks each individual to brainstorm a topic in silence and note down their thoughts (Delbecq, Van de Ven, & Gustafson, 1975). The second stage involves the group members sharing and developing their thoughts through a facilitated discussion, rather like a focus group.

The aim is to encourage all members to contribute and discourage any dominant group members from unduly influencing others. In other words, agreement with someone else's view should not occur simply due to the identity and position in the organization of the individual, but may occur through negotiation and discussion based on an evaluation of the arguments or rationale. The result of the facilitated discussion should be to reach agreed conclusions. In the context of project risk the conclusions may be on the drivers of risk for a project opportunity and on the perceived level of risk attached, though this may require more than one discussion, especially where the technique is also new.

The repertory grid technique (RGT) is a grid drawn up with instances of a phenomenon in the participants' range of experience as the columns (elements) and the attributes (constructs) used to differentiate the elements as rows. It is usual for the participants to select the elements as representative, in this case of projects within their experience, and to generate the constructs by a process of comparison of three elements (triads) at a time (Cassell & Walsh, 2004). Participants start with a focus of investigation, in this case project risk, and are asked to say what makes two out of three elements the same and the third one different and give it a label. The extremes of the construct may also be labeled as the poles of a continuum that elements can be assessed against.

For example, if comparing three new product development projects within the participants recall, one may stand out as having greater novelty than the other two, so the construct may be labeled “novelty” with high and low novelty as the bipolar reference points. Other projects can then be assessed on a scale (say 1 to 5) from low to high novelty. RGT was first developed for use with individuals, and their scores may have been aggregated to find a group assessment. RGT can be used with individuals or directly with groups (Johnson & Johnson, 2002). It is important with group RGT for participants to define each construct to ensure that they have a shared understanding of the term. For use in a project risk assessment framework, where managers will have cause to refer back to the project risk constructs when discussing later projects or during post-audit reviews, it is advisable to capture these in a glossary of terms. Limitations of RGT in knowledge acquisition are discussed by Rugg and Shadbolt (1991).

Cognitive mapping uses a visual representation of constructs, organized around the central theme, rather like a mind map (Eden, 1988). It tends to reflect multiple levels or hierarchies of constructs, whereby detailed or subordinate constructs can be clustered or grouped into higher order constructs, elicited by a process of laddering (Rugg, 1995). Its use in generating strategic options and formulating strategy has developed (Eden & Simpson, 1989) over the last 20 years (Huff & Jenkins, 2002). Analysis of failed projects has produced useful insights into project risk (Williams, Ackermann, Eden, & Howick, 2005), but to benefit organizations in terms of avoiding or mitigating risks it needs to be used in the risk identification process at a pre-decision stage, especially in innovation projects (Harris & Woolley, 2008).

Cognitive mapping can be used with or without RGT, and both RGT and cognitive mapping can be used with or without the NGT. All three methods were used at various stages in the longitudinal case studied here, as part of an action research program involving multiple managers in a single organization over a period of eight years. The research approach taken for this paper is one of critical reflection and analysis of evidence collected over that period from the longitudinal case study and related research. It involves a re-analysis of evidence collected in the first four years, and evidence gained in later years.

The investigation began in 1996 when a new managing director (MD) of a subsidiary of the group was tasked with improving performance in a UK logistics company, where virtually all the business was defined as projects, usually with medium term contracts. In this project-based environment the decisions to invest in a new project were largely initiated by invitations to tender from existing or prospective business clients. The MD considered that the key to improved company performance lay in project selection and was keen to improve the project appraisal process. In particular, the section in the business plans put forward to secure project funding that dealt with project risk was not well written and there was no formal system in place for project risk assessment. The researcher was seeking access to investigate managers' perceptions of risk in project appraisal and prioritized the logistics industry as having frequent instances of strategic project decisions to study. The two aims were combined in an initial action research project which developed a project risk assessment tool later named Pragmatix, facilitated by the researcher.

Within a year or so the MD became group chief executive and the opportunity arose to test out Pragmatix across Europe and assist in embedding it into group-wide procedures with internal facilitators by writing a user manual and providing training for a number of different executive teams (1997 to 1999). This activity led to further research and Pragmatix was updated and extended. The research also investigated the use of a compatible competitive analysis tool, and later expanded to consider post-decision measures of performance, including key performance indicators (KPIs). The majority of new research material reported here was gained through two company-sponsored projects that focused first on the evaluation of Pragmatix by management teams across the organization, including managers who were new to the company (1999 to 2001). The second explored integration of Pragmatix with other aspects of management control. This was named SMART, short for Strategic Management Accounting and Risk Techniques (2001 to 2003).

Data were collected from a number of sources over the eight years, including project papers for more than 40 projects, notes of approximately 50 meetings with individuals and groups, including at least 18 tape-recorded group discussions facilitated by the researcher and feedback from participants at two corporate conferences (1997 and 2000). It is estimated that more than 100 managers in the organization participated in the research. Notes were made at or shortly after all meetings and sent to participants for validation. In addition documentary evidence was collected from newspaper reports, published accounts, and the company newsletters. Files of correspondence have also been kept. This evidence has all been reviewed to reflect on the development of risk assessment techniques, process and thinking in the organization over the eight years, with reference to recent literature. The findings and discussion are organized under the same five headings used in Harris (1999) of project typology, project risk attributes, risk assessment process, timing (pre- and post-decision), and use of the repertory grid technique.

Findings and Discussion

Project Typology

Most projects studied in this case organization could be described generally as business development projects. Some projects involved investing in dedicated facilities like fleets or distribution centers bearing the client's branding/insignia, which were labeled “external client” projects. Some involved investing in central systems, such as pallet-tracking software, to benefit all clients, labeled “infrastructure” projects, and more often in continental Europe, had features of each, as in the case of shared distribution centers, labeled “shared user” projects.

A set of 12 project risks were identified for these three types of business development project, defined by participants in a glossary (Harris, 1999, pp. 369–370). While the managers rated them differently in terms of relative importance for the three types, the definition of terms was sufficiently consistent (agreed by consensus) across types. Definitions were included in the first Pragmatix user guide, written in 1997 and issued in January 1998.

A possible fourth project type investigated in 1998 related to a specialized manufacturing process operation within a subsidiary. It was the only project proposal of a new product development type, involving technical innovation, and was appraised at the research and development stage. As the group had no need to assess other R&D projects, this type never made its way into the Pragmatix user guide. It is therefore included in the findings here as type 5 below, as another project type was later labeled type 4. By January 2003, version 6 of the user guide was issued, including the fourth type of project, “business acquisitions.”

The types of project that emerged from the longitudinal study are summarized as:

1. External client – known by a customer name (market development)
2. Infrastructure – new or updated facilities not linked to a specific customer (for example, an IT project), known by location, business function or software name
3. Shared user – usually a new site or site development, known by a geographic location name
4. Business acquisitions – purchase of assets or shares, known by target company name
5. Product innovation – development of new product, known by product name

These five types of project correspond closely to the types found in a cross-sectional survey undertaken in 2005, shown in Exhibit 1. The table shows how many of the 91 managers in the 65 companies spread across 24 sectors had experience of strategic investment decisions of each type, and then which type they chose to focus on to answer questions about the process.

Compliance and “other” were the two not identified in this case study. The project risk attributes for the five types identified here may be expected to occur in other companies and sectors where projects share common characteristics.

Project Risk Attributes

Exhibits 2 to 4 show the risk attributes shared by the first three business development-type projects and types four and five, business acquisitions and R&D-type projects.

During the development phase there was a lot of discussion about whether strategic fit was really a risk or whether it was more of a prerequisite for the project to be considered at all. It was decided by participants that it was both and hence it remains. Interestingly, strategic alignment is identified as an under-researched aspect of capital budgeting (Langfield-Smith, 2005). It is also an example of a risk factor that can be regarded as either positive or negative when assessing a proposal. Many other attributes included in Table 2 can be assessed as upside or downside risk, including expertise, impact, cultural fit, and market strength. This shows that risks were not just seen as hazards or problems in this organization, but also as opportunities, which was quite forward-thinking at the time.

The fourth risk attribute in the project opportunity cluster, the quality of customer/supplier was added after a review meeting where the group finance director suggested that financial risks were perhaps not adequately covered. There were two areas of concern. One was the risk that a contracting party went out of business before they had fulfilled their contract obligations. There was a credit check undertaken as part of due diligence before contracts were actually signed, but it was felt that those organizational members, for example business development managers, who had face to face contact with customers, would be best placed to assess this risk at the early pre-decision stage. Feedback from others indicated this was reasonable, so the Pragmatix user guide was updated to include this extra attribute in version 4 (December 1999). The second area related to asset specificity of the fixed assets invested in for the project, as many were so client-specific that resale or re-use values built into the DCF analysis could be at risk. It was suggested by others in the meeting that this aspect of risk could be built into the sensitivity analysis in the DCF presented in the project paper, and that it may also be considered within the existing risk attributes of complexity and quality of information, which relate to the planning assumptions.

By 2000 the company had embarked on an acquisition strategy, so acquisition opportunities were very much on the group board member's minds. The earliest business acquisition project had been assessed using the same risk attributes as project types 1 to 3, which did not fit ideally well, thus the fourth project type was developed. Exhibit 3 shows the risk constructs that emerged from the group discussions in tabular form. This time there was more discussion during the elicitation stage around the inter-relationships between the risk factors, which were noted for inclusion in the glossary. The brief definitions are summarized from the glossary in Harris (2007, Table 2).

The full glossary of terms included cross-references between constructs where participants found it difficult to have a conversation about one risk factor without referring to another. This is easier to present in the form of a cognitive map (discussed later). While the definitions of constructs 1, 2, and 8 relate specifically to the logistics industry, they could be redefined and the other constructs could be understood and applied to other industries. The focus of construct 9 was on the target company, but being new to managing business acquisitions, this was also a major risk for the bidding company, which with the benefit of hindsight, the chief executive acknowledged during a post-project review meeting in 2004.

The project risk attributes, previously unpublished, for the innovative new product development were developed after the management team concerned with the project had attempted a risk assessment using the existing grid developed for type 1 to 3 projects. In common with many research and development projects there were multiple decision points, and the following risk constructs were elicited after the product had been designed and technology had been tested, prior to a commercialization decision (Exhbit 4).

This particular project never reached commercialization, as legal issues proved insurmountable, and no further projects of this type arose during the period under review. Hence this type was not formally adopted in the Pragmatix user guide. The risk constructs may be sufficiently generic to re-use in other situations and match reasonably with those found in other product development case studies (Harris & Woolley, 2008). The two cases reported by Harris and Woolley were named “flavors” and “carotene,” and had similar project characteristics to the project evaluated in Table 4. “Flavors” had five clusters of constructs labeled business model, timing, market requirements, product viability, and technology. “Carotene” had four clusters of constructs, labeled capacity, market response, scale up, and givens (product specification). These fit well with corporate factors, especially constructs 2 and 3 (capacity and expertise), competitive advantage factors, especially 13 to 15 (attractiveness to market) and project factors, especially 5 (volume) and 10 (quality of information). The “carotene” team identified consistency of product quality when entering mass production process, which would also have been highly relevant in the project in Table 4.

Risk Assessment Process

Once risk attributes were established, each new project opportunity was assessed using this framework, as set out in Harris (2000) using a grid that combined a perceived risk score from evaluating the level of risk attached to the new project with weightings that represented the perceived priorities or importance attached to risk factors for projects of its type. This style of scoring has been used in the application of repertory grid techniques (Fransella & Bannister, 1977) and in cost benefit analysis (Huber, 1974). The project risk assessment grid proposed here may appear to be similar to the traditional PIG (probability impact grid), as the raw score may be interpreted as an assessment of the likelihood that this factor will impinge on the project, and the weighting indicates the relative importance of the risk attribute, which could be interpreted as the impact this factor could have. However, the value of this technique lies in the qualitative approach to risk identification, resulting in an improved understanding of the risks through discussion. An example of a project risk assessment grid is shown in Exhibit 5.

This example is the result of an assessment exercise facilitated by the researcher in June 2000. It was assessed as higher than average risk overall (3.59 on a scale of 1 to 5). The risk scoring system is based around 3 being the normal level of risk attached to projects of this type in the assessors' experience, 2 is low risk and 1 is very low risk (or upside risk), 4 is high risk and 5 is very high risk (downside risk). This project, with an overall score of 3.59, is therefore assessed as above average risk for projects of its type. However, more important than the overall score is which risk attributes are contributing the most to that score, in other words the highest risk factors.

In this case it is expertise and complexity, both at a maximum of 25, followed by size at 20, then planning timescale, demands of customers and proposed contract terms all at 16. Next in interpretation of the grid is the rationale behind the scoring for the highest risk factors. For this logistics project the high score for expertise was due to a lack of availability of skilled drivers in the area, and for complexity, the number of assumptions and pattern of drops. As the contract represented approximately 35% of revenue for the relevant business unit, size was high risk due to the earnings at risk if the customer's demands were not met. These reasons were noted in a comments column on the spreadsheet to the right of the scores.

Feedback on the use of Pragmatix was obtained from a number of management teams during 2000–2001 and two sample evaluations are included in the appendix to this paper. It could be argued that many of the strengths and benefits identified could also be achieved using a standard risk PIG, but some might relate specifically to Pragmatix that may not be achieved with a standard PIG. A comparative analysis of the two approaches is shown in Exhibit 6.

Perceptions of risk vary between individuals, which is why the group needed a consensus-seeking process to explore any differences and reach agreement. Perceptions of risk also change over time, especially when the live project proceeds and new information comes to light. This lack of temporal stability is explored further in the next section.

Timing of Risk Assessment

For business development projects of types 1 to 3, a process model of strategic investment appraisal was captured (Exhibit 7).

This depicts a bottom-up process of identifying project opportunities in a divisionalized organization, where the ideas are generated and defined at the business-unit level. With invitations to tender (ITTs) for most logistics business this was invariably the case. The diagram implies that the group board's first knowledge of the opportunity was after a full business case had been worked up and agreed at divisional level (stages 2 to 5), immediately prior to the decision to fund the project or not (stage 6). In fact, divisional managers were required to alert the board to significant projects at the early screening stage (stage 3), where the score for size in the early risk assessment could trigger such reporting.

While training and development exercises often assessed projects at a later stage (to ensure adequate knowledge of the risks), the user guide advocated its use at stage 3 to achieve the most benefits (reflected in the feedback). Teams were then encouraged to update the grid (both scores and narratives) before submitted the case to the board at stage 6, and to review it at stage 7 as part of the post-audit process. Where used at least 3 times in the projects development, the benefits of Pragmatix as a dynamic tool could be fully realized. However, some managers were uncomfortable with the notion that the level of risk could change so much over time, and the variances served to emphasize the subjectivity of the exercise. This was addressed by further workshops that explored the reasons for changing the assessment, when new or better information emerged.

In the latest version of the Pragmatix user guide, issued in 2003, the risk attributes were re-clustered, removing the labels corporate, project, external, and competitive position, and replacing them simply with two headings, pre- and post-decision (Exhibit 8).

This reflected the outcome of group discussions in the SMART project around controllability. In order to integrate Pragmatix with other business systems and embed risk assessment in the business planning cycle (Exhibit 2), it was agreed to split those risks where the opportunity for mitigation runs out when a contract is signed and the company committed, from the ongoing risks that require monitoring and action as the project is implemented. The risk constructs were also revised to reflect the feedback gathered from 2001–2002. The action plan to improve Pragmatix was recorded as:

1. To delete the first attribute “strategic fit,” as it caused a lot of debate for very little value, since all projects being assessed were being given low scores as they were all considered to be a good fit with the strategy (or they would not be considered). It was not an operational construct, but a corporate construct that had served its purpose as a group discussion prompt back in the days when the strategy was less clear. It was difficult to verify its inclusion based on current risk literature.
2. To focus more attention on the key operational risk areas by changing the order and clustering of the remaining risk attributes.
3. To identify a smaller number of the risk attributes of importance post-decision and use these to build a more formal link with KPIs and to the project review process for monitoring. For example:
   • Expertise (action taken to identify and remedy shortages?)
   • Customer relations (managing the demands of the customer)
   • Environmental factors (seen as more of an amalgamation of possible operational risks that might require mitigating action to minimize the effect, even where the risk was seen as low overall)
   • Quality of information (where this can be weak prior to the decision point, it may include assumptions that need revisiting in order to manage the project within budget)

The ongoing monitoring of some risk attributes could help to alleviate the problem identified at the last meeting of over-reliance on capturing project risk in a single number (weighted score).

This resulted in the 10 risk attributes grouped into 5 pre-decision and 5 post-decision, shown in Exhibit 8. The other “casualties” in addition to strategic fit were image/reputational impact and cultural fit. These were rarely assessed as high risk, had medium/low weightings, and were seen as either unimportant or even diluting the overall risk score.

Use of the Repertory Grid Technique

This technique combines risk elicitation with risk assessment in a single technique. It can be used with a nominal group technique to use it as a group process rather than at individual level, so long as Kelly's corollaries are tested positively. In the case study it has underpinned the development of risk assessment in an organization that previously had no formal risk assessment structure or method. After eight years of use it has been refined, maintaining it as a group activity. Its strength lies in its roots in PCT, as risk assessment is too subjective to rely solely on mathematics. It offers a qualitative method of knowledge acquisition that can be used to elicit and share intuitive feelings about projects and risks within a structure that allows metrics to be applied without need of probabilistic calculations. It requires a trained facilitator who will usually be external to begin with, but internal facilitators can take over.

It can be combined with cognitive mapping to benefit from a visual representation of project risk. This was tested for type 4 business acquisition projects (see Harris, 2007, p. 1069).

Key Insights and Implications

Five of the seven most common types of strategic investment decision were studied in this organizational context. While some of the project risk constructs may be specific to the industry context, for example strategic location and familiarity with territory (Exhibit 3) in business acquisitions, though this could apply to industries other than logistics, at least 60% are generic strategic risks faced by many organizations in any industry. Some project risks are specific to the type of project, for example, the competitive advantage attributes (Exhibit 4) in new product developments, or the integration with existing business, management ability, and method of acquisition (Exhibit 3) in business acquisitions. Detailed understanding of project risk attributes and the specific terms used may vary across individual managers even within the same organization, so gaining a shared understanding of the risks and capturing that understanding in a glossary of terms is vital for risk assessment to be meaningful and sustainable.

Over the eight-year study, key project risk attributes changed very little. In 1996–1997, there were 12 project risk attributes agreed for the first three project types. By 2003 one had been added to reflect financial risks attached to large contracts with new or existing customers, and three had been dropped as no longer important enough to differentiate high from low risk projects (strategic fit, reputation, and cultural fit of parties), though a new definition of cultural fit (compatibility of corporate cultures) was retained for business acquisitions. The focus of discussion on project proposals had shifted from whether it was a good strategic fit toward how risks could be mitigated and monitored post-decision. This may be interpreted as demonstrating a more mature managerial stance, where the confidence that the strategy was agreed and clearly communicated reduced the uncertainty over whether a new project fitted with that strategy.

The focus on mitigating risks and the split of project risk attributes between those actionable pre- and post-decision indicates a shift toward more “joined-up management,” whereby risk assessment was becoming more embedded in management thinking. Full integration of information systems had not been achieved, but the organization was at least attempting to improve knowledge management by sharing the outcome of project appraisal with operational managers tasked with project implementation.

This longitudinal study has many facets worth further reflection, but the focus of this paper is to demonstrate how methods drawn from psychology were employed in such a way as to develop a risk assessment tool that managers could apply at a strategic level. Capturing the insights and intuitive aspects of managerial judgment in this way does not negate the need for measurement of possible outcomes and probabilities of risk impacting on the project, but it helps to promote dialogue and debate around the issues subject to uncertainty such that risk management may become more meaningful than a purely numeric exercise. It is recommended that to maximize the benefits from investing managerial time in careful elicitation of project risks, this is captured in sufficient detail that the learning can be carried through the project life cycle. Thus the broad areas of risk identified pre-decision that can be controlled post-decision (residual risk) entered into the project risk register, where ownership and action can be recorded and monitored. The methods used here have demonstrated how an essentially qualitative approach can improve risk management practice.