Identifying and acting on early warning signs in complex projects

Abstract

This paper considers how project assessments may be used to identify early warning signs in projects, and how complexity influences this. This study confirms the main findings in earlier literature and gives an overview of current documented practice. Then comprehensive empirical research is performed to expand this. We find that project professionals are not very good at detecting early warning signs and even less good at acting on them. Barriers that lead to this are identified. Identifying and acting on early warning signs is particularly important in the initiation or set-up phase of the project, but also relevant in later phases, including execution. The nature of early warning signs and the way these can be detected change over time and with the evolving situation. We see that project assessments, typically performed as part of gateways, are useful in identifying early warnings connected to the documentation and formalities of the project. As the complexity increases, the assessments have more limited use, and the project is increasingly dependent on detecting early warning signs by informal “gut-feel” approaches. This indicates the increased importance of knowledge, experience, and communication skills in complex situations. The paper concludes with a list of early warning signs.

Introduction

We are often surprised at how projects turn out. Complex projects often do not behave in the way we expect, and in particular, effects within complex projects are often time-delayed and take time to emerge. Project assessments (however called) are carried out worldwide, often to evaluate to what extent the project documents are developed in accordance with expectations and formal criteria and to support decision making; these have to look through the complexity of a given project in its context and identify the relevant early warning signs of project problems, project failure, under-performance, or cost overrun. Experience and current literature seems to indicate that we are not very good at picking up early warning signals.

This paper reports a study of how project assessments may be used to identify early warning signals in complex projects (Klakegg, Williams, Walker, Andersen, & Magnussen, 2010). This research represents a project owner's perspective. The study includes analysing guidelines, interviews and cases from three countries: Australia, Norway, and the United Kingdom. The focus is on well-established industries with traditional investment projects including the development of some form of physical or digital infrastructure, typically construction, energy, oil and gas, telecommunications, and ICT.

The study included a literature review as a starting point and this confirmed that we are, in general, not very good at picking up on early warning signs. In particular, the literature points out problems related to three areas: understanding project risk and uncertainty, comprehending complexity, and detecting people's tacit knowledge and understanding of how they respond and interact. There is no doubt, however, that project professionals and academics in the field of project management research do try to improve the current practice by introducing new and improved methods and tools in areas like risk management, project planning and control, as well as project governance. There is an abundance of literature discussing aspects of how to identify uncertainties in the project development, unhealthy conditions that may lead to problems later, and why project do or do not perform as expected, as well as advice for improving performance of these aspects (for an overview, see Klakegg et al. (2010)). This paper does not take up this line of inquiry, but instead reports the empirical part of the study.

This article reports on a comprehensive exploratory fieldwork designed to look at how project owners actually install regulatory frameworks with project assessments and discuss some aspects of the outcome. The owners’ attempts to strengthen the project governance are done with the ambition to identify early warning signs and act on them in order to secure successful projects. The authors take a closer look at these practices to find out how successful they are.

First we looked into a selection of nine (public and private sector) governance frameworks to find out how frequently project assessments are made and what the guidelines prescribe as good practice. This part of the study comprises document studies and interviews with key experts on these frameworks. To investigate the actual practice and to collect empirical evidence we conducted semi-structured interviews with fourteen experts, looking at what companies and public entities do to implement project assessments, focusing on methodological choices and the effects of established practices (i.e., what they look for and how they utilize what they find). We followed this up with the analysis of eight case studies to find out what the project assessments had identified as early warning signs and whether these had actually been confirmed in the time after assessment (i.e., whether the right things were looked for and action taken). The cases were chosen to represent the same categories of settings as used in the interviews to secure consistency in the material.

In this article, we choose to report specifically on our findings concerning the ability to identify early warning signs and act on them. The basis comes from research into how methods are designed to identify warnings, and why they fail to detect early warning signals, how project assessments can handle the complexity in projects and their context, which practices seem more appropriate in which contexts, and how identified warnings are acted upon.

Conceptualization

Early Warning Signs

An early warning sign is an observation, a signal, a message, or some other form of communication that is or can be seen as an expression, indication, proof, or sign of the existence of some future or incipient positive or negative issue. It is a signal, an omen, or an indication of future developments (Nikander, 2002).

Ansoff's (1975) ideas of responding to “weak signals” state: “A firm that wishes to prepare for strategic surprises has two options. The first is to develop a capability for effective crisis management—fast and efficient after-the-fact responsiveness to sudden discontinuities…The second approach is to treat the problem before the fact and thereby minimize the probability of strategic surprises…Both approaches deserve management attention.” This paper has a focus on early warning signs (EWS)—the proactive approach.

Loosemore (1999; 2000) identifies three types of crises in a construction project management (PM) context. Perceptions of an impending creeping crisis present EWS that are understood but unaddressed until the crisis occurs. Sudden crises occur seemingly without warning whereas periodic crises occur in cycles that may or may not be understood. Many crises appear without accompanied contingency plans—often being perceived as low probability but high potential impact—events perhaps best tackled using an emerging strategy and having sufficiently skilled project management teams to recognize EWS and react appropriately (Mintzberg, 1987; Mintzberg, Ahlstrand & Lampel, 1998).

This study, however, focuses on a project's front-end because establishing sound governance at this phase is so pivotal (Williams, Samset, & Sunnevåg, 2009). Accordingly, we also focus on project management governance frameworks because they are closely linked with project assessments that reveal EWS. A study of three existing frameworks in the UK and Norway (Klakegg, Williams, & Magnussen, 2009), showed how project assessments that are embedded within the governance structures operate effectively to identify EWS and address emerging problems. Detection of EWS can be appropriately designed into the delivery system. Balachandra and Raelin (1980) presented a model indicating that project success factors could be used for developing an EWS model—an approach supported by Sanchez and Perez (2004).

Types of EWS

Traditionally, project performance measures are lagging indicators—consequences of activities and incidents—not leading indicators that can provide more relevant and valuable information. The basic idea of EWS is to focus leading indicators; the question is where to look. Kappelman, McKeeman, and Zhang (2006) indicated that people-related and process-related risks scored higher than product-related risks as dominant EWS of IT project failure. Syamil, Doll, and Apigian (2004) argued that behavior-related performance measures evaluating project process are mediating variables affecting the extent to which the given process contributes to the overall project result. Hoegl and Weinkaug (2004) found that collaborative processes during a project have predictive properties in regard to later team performance and can serve as EWS.

Cultural or disguised human EWS should not be overlooked. Understanding human gestures and expressed metaphors about lived experiences provides an important window into feelings and emotions (Whitty, 2010). These reveal many subliminal and hidden human EWS to project managers having the emotional intelligence to sense unease (Dulewicz & Higgs, 2000). Often a workplace culture suppresses people's ability to express such fears. Unease can be measured through stakeholder engagement tools (Bourne & Walker, 2006) or surveys, particularly well-constructed employee feedback surveys (Lloyd-Walker, Lingard, & Walker, 2008). Providing anonymous access to whistle-blowing procedures (Beauchamp & Bowie, 1997) is now becoming more common as part of a response to victimizing in the workplace (Dessler, Griffiths, & Lloyd-Walker, 2007: p. 128).

Interpreting human behavior is always a challenge. Nikander and Eloranta (2001) and Nikander (2002) present an extensive list of types of signals including “gut feelings” and “non-verbal information” as well as “differences and deficiencies in project culture” and “miscommunication” that identify potential problems. Recent work by Whitty (2010) discusses body language and cues that people naturally use to describe project management processes and how they feel about their project experience.

Project Assessments

Project assessments are a wide concept comprising all types of appraisals and examinations of project documents and practices in order to support decisions, learn from experience, or assure that expectations or formal criteria are met.

The main types of project assessments are summarized in Exhibit 1. Oakes (2008) categorizes assessments by frequency/formality, the type of review team (independent specialists or peer reviewers), and the focus of the review (business or technical). Assessments can take place at all stages of project development.

Exhibit 1: Summary of the main types of project assessments and associated references.

Our study incorporates the use of a standard project model, including phases and development stages, similar to those used in general project management literature as a reference. One typical element of project models with an owner perspective is “gateways” or “stage gates.” Literature on the stage gate approach and how it aims to pre-empt potential problems that make a project nonviable is well known (Cooper, Edgett, and Kleinschmidt, 1997; Cooper, 2005; Office of Government Commerce, 2007), however, as Flyvbjerg and others caution (Flyvbjerg, Holm, & Buhl, 2002; Flyvbjerg, Rothengatter, & Bruzelius, 2003) overoptimistic assessments of benefits and underestimates of risks can subvert this process as a way of flagging a possibly unsustainable project. The front-end in a project life cycle is an effective time to look for early warning signs, but often these are purposely overlooked, as Flyvbjerg et al. (2002) suggest, or they simply are not envisaged. Later, during project delivery, periodic reviews may assess risks, progress, and indicators of development. Sometimes that occurs in response to identified potential or real problems through audits, health checks, or benchmarking exercises. This stage of a project's life cycle involves intense and distracting activity and it is easy to miss EWS. Additionally, some actions may be too late to avert when the warning signals are acknowledged; the original warning signals may have been conveniently ignored or intentionally hidden.

Some assessments may be performed by the project planners and responsible executors themselves, but controlling and questioning one's own work obviously has its clear limitations. To strengthen the independence of scrutiny and credibility of the findings, the assessors are normally more or less independent of the project organization. They might be internal to the organization but are often all external experts.

Focused review systems and monitoring that can detect early warning signs through a governance structure is an integrated element in project management practices. Post-project reviews need to be comprehensive and context rich (Williams, 2007). Post mortem analyses of unsuccessful projects also show that there are evident EWS well in advance of the final failure.

The Influence of Complexity

Intuitively we would expect complexity influences the possibility to identify and act on early warning signs. Complexity includes structural complexity (Baccarini, 1996) and uncertainty (Turner and Cochran, 1993). There is a large body of literature on project risk management, for the “known unknowns.” However, much uncertainty comes from lack of a clear unambiguous goal (Linehan and Kavanagh, 2004; Engwall 2002). Even when the goal is known, moving towards it can be a messy uncertain process, as participants “make sense” (Weick, 1995) of the project and work towards project delivery. Problems within projects are often subjective and interpersonal, resulting from a team of people working uncertainly towards an uncertain goal with emergent complex team behaviors (e.g., Stacy (2007)).

Understanding the outputs or behavior from the input effects is difficult in assessing complex projects. Simon (1982) defines a complex system as “one made up of a large number of parts that interact in a non-simple way.” The New England Complex Systems Institute (2009) states “The study of complex systems is about understanding indirect effects.” The causal relations between early indications or incidents and later results are seldom obvious, and often very complex. Much project complexity comes from the human-oriented social aspects—projects have “behavioral complexity” as well as “dynamic complexity,” making them “wicked” (Roth & Senge, 1996). Hence, as well as internal complexities, such as technology and interfaces to existing systems, external complexities such as stakeholder relationships (Pryke & Smyth, 2006) bring particular difficulties in understanding project behavior. Williams (2005, 2007) discusses how to draw lessons post hoc from complex projects and the difficulty of this in predicting the behavior of complex projects; the difficulties in predicting mid-project behavior are even more apparent.

Guidelines for Project Assessments

The study looked at established governance frameworks and their guidelines to consider what is considered good practice and to check whether these specify what to look for in terms of early warning signs. Nine frameworks were studied, representing both public sector (six frameworks) and private sector (three frameworks) in Australia, Norway, and the United Kingdom, and representing a wide variety of approaches, methods and structures, some general, some complex, some control-based.

The study report (Klakegg et al., 2010) presents examples of the governance frameworks and the use of project reviews incorporated in them. We found that these frameworks are very common and that project assessments, in some form, are obligatory in most of them. They are also reported to be effective. We found some differences between sectors: In the private sector, the frameworks tend to be more internal and integrated with other systems, and the decisions attached tend to be assigned to a smaller number of people. The division between governance and management is often not so clear in the private sector. In the public sector, there are normally fewer stage gates and a more external procedure. Governance and management tends to be more clearly divided.

Roots of problems in later project phases are found in processes and decisions at the front end of projects. Issues that arose in the study of governance frameworks included:

• Issues of public policy and market shaping (Office of Governance Commerce, 2010a)
• Initiation documentation and the business case (Office of Government Commerce, 2010b) or not being able to answer the fundamental questions:

• What is the project aiming to achieve?
• Why is it important to achieve it?
• Who will be involved in managing the process and what are their responsibilities?
• How and when will the project be undertaken?

But it remains a moot question whether the project reviews are able to detect the important EWS. It is obviously a fundamental assumption that they do. Several guidelines mention detecting early warning signs as one of the purposes of assessments. The owners of these frameworks are clear in their judgment: they report that these frameworks are successful and deliver results as intended. Our research indicates this is true, but also that the assessment approach has limitations as to what kind of EWS these frameworks are able to identify; and whether the results can be considered early warning signs is a question of whether they can be detected early enough. The framework guidelines also do not generally specify for which early warning signs to look.

Assessments in Actuality

We interviewed fourteen respondents, evenly distributed among the three countries, about their experiences of governance frameworks, assessments, and early warning signs in order to get closer to the reality behind the current practices. We provide details of those interviewed and our rationale for the study elsewhere (Klakegg et al, 2010).

Context and Experience

First, we asked about the definition of early phase. This definition varied across the organizations interviewed, with a differing degree of specificity in the definitions, ranging from more informal boundaries to more concrete points in time. The end of the early phase seemed more pronounced, marked by formal approval at predefined gates.

Regarding the use of gateway and phase models, most private and public sector organizations had (more or less) clearly defined gateway models, but some did not, and there seemed no pattern to predict which type of organization would have implemented one. The models ranged in extent, from only two gateways in some public sector regimes to five, six or more in some public and private sector models. Transition from early phase to project was in almost all cases dependent on a formal approval or decision by someone mandated to sanction that transition, such as executive management, accredited reviewers, the client, or politicians. Internal resources were usually involved, but with a strong presence also of external resources, typically consultants or other types of advisors. There seems to be a tendency for external resources to be used more when the gateways are considered formal ones.

Current Practice in Project Assessments

All organizations had a range of various assessments, applied in all of the project phases throughout the project life cycle. Areas addressed included:

• Stakeholders, through more or less formalized analyses.
• Political processes and understanding that led to the decision to pursue the project.
• Technical viability and “deliverability” of the project.
• Risks and opportunities, perhaps the most obvious source of early warning signs as risk elements are per se issues that could cause problems and therefore need to be monitored.
• Resources to perform the project.
• Cost estimates and “affordability.”
• Business case development.
• Environmental impact analysis.
• Benefits and “value for money.”

Documents supporting the assessments included technical and financial documentation, reports from external consultants, and different checklists (the documentation in each case depending on the issue in focus in the actual assessment). Some interviewees mentioned that assessments also relied heavily on interviews and/or discussions with key people. Common approaches to executing the assessments include group discussion sessions, interviews with individuals, peer reviews, observations in meetings, and completing checklists.

Identifying Areas for Early Warning Signs

The main impression was that although many attempts were made to learn from previous projects—this was rarely very effective. The reasons cited ranged widely: no time to prepare lessons learned, reluctance to “air dirty laundry,” projects are viewed as unique so one can learn very little from past projects, and reports that are short enough so that people will read them lack sufficient information about the project context to enable real learning. One respondent summed this up: “There are many lessons identified, but not very many learned.”

We did, however, learn of some practices that might help make lessons learned more useful:

• Consistently writing down insights understood during a project to ensure remembering them;
• Ssenior managers appointing project managers and indicating which previous projects (and thus lessons-learned) might be relevant;
• Having to review at least three similar projects when new projects are being proposed for execution to assess whether lessons learned apply to the new project; and
• Bringing in external assessors because they see many projects in many different organizations and represent an efficient means of experience transfer.

There were several suggestions about how post-project lessons learned can be used to identify early warning signs. Reports can contain information about problems experienced that could be used for early warning in the new project. The likelihood of picking up on such issues appeared to increase if lessons learned were presented orally. Access to corporate expertise was mentioned as important. Bringing in external views into the project group is helpful. Converting lessons learned and/or post-project reviews into either specific checklists of possible problems or more open lists of possible areas of concern can provide new projects with direct sources of early warning signs.

When asked about early warning signs of complex projects, in addition to “hard” issues such as technological development, the interviewees also mainly identified softer issues such as culture, the lack of an outsider's view or perspective on the project, anchoring in the permanent organization, lack of consistency between stakeholders’ ambitions, certain organizations promoting one solution and trust, as well as more “gut felt” signs, such as detection of unrealism, lack of clarity of thought, or misalignment between quantitative risk analysis and qualitative risk assessments. Contextual factors were mentioned as an important source of complexity, including:

• Location decisions and complications arising from such decisions
• Leadership issues
• Quality of information and documentation produced
• Whether guidelines for early phase assessments and “behavior” are followed
• Relevance of the proposed solution compared to needs
• Culture, and whether specific conditions exist that will make cultural aspects a factor
• Need for development of new technology
• Main risks identified
• Lack of competence in the project team
• Sponsor with unclear expectations and role.

Unexpectedly, these factors are mainly not really contextual: they are about the decisions and actions performed within the project or organization. It may be that the respondents are closer to these factors than the really contextual ones (a warning sign in itself), or the respondents may have focused on projects’ responses to the really contextual factors, thereby focusing on what projects can influence.

One interviewee in particular pointed out the assessor's role and ability to see the early warning signs. Some people are focused on details, while others are preoccupied with the bigger picture. The ability to reflect on what happened in a project is crucial, for sponsors, project managers, team members, and assessors; and this requires both ability and time.

Early warning signs might arise in the interaction of issues and problems (“knock-on effects”). A few respondents (private and public sector) confirmed that such interaction is often seen (“issues tend to come together”), particularly if one overlooks a problem as some other issue may come up that combines with one's already unsolved problem to make things worse. “Knock-on effects” can also be caused by “silo thinking,” whereby different parts of an organization work toward their goals, not understanding that their actions influence other parts of the organization, ultimately causing different problems to combine. Respondents were not clear why such knock-on factors are hard to identify, except that it is difficult to provide project managers with enough time to think ahead about such potential problems, and we need to make people in projects question the assumptions.

The Use and Usefulness of Early Warning Signs

We looked into the use of previously identified early warning signs at a later stage in a project. One public sector organization said that such signs change so much over time that they end up not being used very much. This is in sharp contrast to a private company that has implemented a balanced scorecard approach including early warning signs as one set of indicators, assessed in regular meetings which are held at intervals of several weeks. Another private company employed a similar approach based on “traffic lights” for risk elements identified as early warning signs, again reviewed regularly. Three project management consultants all concurred that EWS that originate from external project assessments are “stronger” than those based on internal ideas, both because the external assessments carry more weight and because the sponsor will be aware of them. One consultant said that such signs are most effective if they are included in the project reporting system, that is, reports to the stakeholders must include an assessment of the status of problem areas expressed in early warning signs. One private company had tried to institutionalize common early warning signs and lessons learned by modifying the stage gate requirements to ensure that such issues were properly addressed.

We also asked to what extent insights gleamed from undertaking various assessments enabled identification of future problems in both the project and the project's business setting. Answers were not conclusive; a private sector project owner claimed the analyses and assessments made were sufficiently good in that no major surprises turn up later in the project (minor issues could occur, but never caused the project to run “beyond 110%”). Two project management consultants, on the other hand, posited that such assessments are currently not good enough, although they have improved over the years. In particular, there seems to be a guiding framework or checklist missing which would aid this work, and there seems to be a natural tendency to focus on the solutions and contents of the project in the early phases (engineers focus on technical issues and economists on the business issues, but no one focuses on the project execution).

A key issue in this study is to examine how useful early warning signs (EWS) are in foreseeing problems (and enabling action on such predictions). At least five respondents (private and public sector) claimed that the use of EWS had been useful in preventing problems, with one of the private companies saying that the performance of their projects had gradually increased over the last two to three decades, in part due to their use of balanced scorecard early warnings. Others were not as unequivocal; one private company said reviews of early warning signs can detect problems, but cited an example where the review came three months later than it should have. One respondent from the public sector said that EWS are often not well articulated and, further, that the most difficult part is interpreting the EWS that occur—in some cases, with hindsight, it is realized that the signs were actually picked up, but were not acted upon. This was echoed by another public sector interviewee, stating that their organization was good at identifying early warning signs, but poor at letting the signs affect decisions. Especially in the cases where the early warning signal is a “sense of uneasiness,” it is difficult to induce action. It can be very difficult to justify the feeling, thus people are reluctant to report it, even if the feeling later proves to be true and something is in fact wrong.

With some dissension in terms of the usefulness of early warning signs, we looked into what causes the less-than-hoped-for effects. The respondents provided various explanations:

• Overly ambitious plans—these are very difficult to detect.
• This is similarly true for the development of new technology and associated difficulties.
• Even when early warning signs are picked up, projects are very difficult to stop. Even when the concerns are documented, the projects’ response is usually to assure that things will be alright and that they “will run even faster,” thus effectively countering warnings.
• Especially with complex projects, it is difficult to identify all relevant early warning signs; hence, the problems that do materialize are things not covered by them.
• People involved in governance discussions and high-level project management discussions have often become too senior to have recent and relevant experience from operational matters, consequently fail to address these issues in early warning signs.
• A tendency for group thinking, where ideas novel to the team's collective thinking and experience will not surface.

Given that there are problems in utilizing early warning signs effectively, we finally asked what could be done to remedy this situation. One private sector respondent said that more discipline was needed in actually using the early warning signs once they have been identified. Another said that a too “heavy” process for identifying early warning signs could be an issue; it stifles creativity and thus fails to uncover all relevant warning signs. A public sector suggestion was to repeat relevant project assessments and the exercise of finding early warning signs several times throughout the project. The project management consultants pointed to the need for a formalized process for finding early warning signs, asking the right questions, and bringing in people with the right competence in the process, including someone “thinking outside the box.”

Analysis of Case Studies

We looked in depth at eight cases, spread across a number of domains and in three countries (see Table 2), to see how early warning signs were detected in practice, which particular signs were noted, the usefulness of the detection systems, the effects of complexity, and how the early warning signs were used in performing these projects. Below we analyze some aspects across all these case stories. Points taken from the case studies are identified in the text by the reference shown above in square brackets, for example, “[N1]” in Exhibit 2.

Exhibit 2: Overview of the case studies.

The selection of eight cases in three countries has enabled us to cover many dimensions because we have been able to look into projects from different industries, both public and private sectors, and with different degrees of complexity. While it is perfectly normal that projects are dissimilar—most project definitions state that projects are unique and temporary—these projects are not only difficult to analyze because of their different sizes, complexity, and task uniqueness, but they are carried out against highly differing contextual backgrounds and different regulatory regimes. Clearly, a summary of lessons learned from such cases have to focus on the most important and those that seem to be relevant above the level of the single project.

About Project Reviews

Generally, the case studies showed the reviews to be useful. Some examples: Major external reviews are useful in providing reassurance and for giving projects stronger legitimacy [N4]. [U2] showed the usefulness of an “open attitude where partners brought out all their issues” attitude. It also particularly showed the usefulness of partnering workshops. Assessments are an element both in developing and identifying the need for new knowledge and working practices [N5], although the transfer of knowledge into the permanent organization is more difficult [N5].

However, reviews need to be well focused: While in [N4] stakeholder assessments to identify needs and priorities dominated the beginning, some misdirection of attention was identified. [N2] in particular, warns against a focus in uncertainty analyses on “obviously” important issues such as detailed technical matters, construction market, progress, etc, which mirror already identified problems, but which do not identify weak early warning signals of future problems and might not actually be the most important. In [N4] also, technical assessments dominated the engineering and construction phases.

What was Good Practice or Practice that Seemed to Work?

Organizations and project teams approached the task of detecting early warning signs in different ways. There were a number of examples of good practice, or at least practice that seemed to give good results. In general, the formal exercises were found to be useful. [U2] found the organization's gateway process useful. [N1] found the early warning signs exercise useful—but it was the exercise itself rather than the indicators. [U2] demonstrated that problems can occur following a stage gate if a project proceeds after the gate with an unresolved issue. [A1] found that a “quarterly program of works review” of all projects allowed for decisions to kill projects—which avoided the situation whereby a project running adrift (with early warning signs pointing to failure) would be left to continue despite feelings of unease by many of those delivering the project. [N3] made the important point that efforts must be made to revise early warning signs frequently and keep them “fresh” in people's minds.

But rather than—or as well as—formal assessments, dialogue is key. Everyday communication and the work situation are better at identifying early warning signs than assessments [N2]; building trust and good communication are good alternatives to extensive use of assessments [N4]; warnings resulting from detected early warning signs were initiated through dialogue [N4]; and in particular, more important than assessments as a source for early warning signs was the dialogue with the stakeholders and the technical observations during a parallel development process [N5]. Indeed, interestingly, “project participation satisfaction” became a valuable “thermometer” of the climate in the project [N1].

Outside the project team, organizational culture is important. Encouraging a culture where the eagerness to develop projects and acquire contracts is balanced by critical assessments of the projects’ viability was seen as important [N3], as was the idea that an appropriate organizational institutional support that gives authority enhances the likelihood of trust and commitment [A1]. Moving beyond the immediate organization, competence and knowledge about relations and stakeholders grow more important as the project's criticality increased; the project team's ability to master these relations became essential to project success in [N5]'s particular situation. Moving from a technical approach via project management approach into relational and politics approach was seen as valuable [N5]. Indeed, in a highly political environment, playing actively in the political arena and being a strategic actor was seen as appropriate [N5] and developing a collaborative culture across organizational interfaces.

Which Early Warning Signs Were Detected and/or Acted Upon?

Generally, all of the case studies (not surprisingly) failed to pick up some of the early warning signs. But some particular points which arose in the case studies can be summarized as shown in Table 3. The important points indicated in the table are the differences between stages of development.

Exhibit 3: Important early warning signs by time period.

Some additional observations:

[A1] shows the fundamental mismatch between project life cycles and strategic planning life cycles. This means that business might insist upon a year-on-year cycle of resourcing and project sanctioning when projects are likely to stretch over several year-end periods. In this case, the initial weakness of this mismatch was recognized, and from 2009, a 5-year strategic planning cycle was adopted.

The sponsor is also important, with sponsor abrogation of responsibility for maintaining focus and interest in the project [A1] and poor quality support from the sponsor to the technical team during project development [A1], both being early warning signs of problems. Also, the qualities of the team provides early warning signs, whether through poorly synchronized technical and business knowledge [A1], poor quality technical support while the application is bedded down at the implementation stage [A1], or technical or business capability maturity mismatched or lower than required [A1].

The case studies clearly indicate that reviews and decision points (stage gates) are useful. In [N4], first bids showed the cost frame would be exceeded. This led to changes in the concept, and a reconsideration of contract strategy. An extension of the bid-consideration period was initiated before commencing.

Barriers to detecting early warning signals identified

Certain possible barriers to the detection of early warning signs were identified. Flyvbjerg et al. (2002; 2003) pointed out that optimism bias in both underestimating costs and overestimating benefits is an important barrier to taking in early warning signs. Optimism will generally make the actors de-emphasize possible warning signals [U1] [N2]. Optimism also occurs in the external client [N3], and indeed in a helpful contractor [N3]. Other barriers are organizational. These include organizational complexity in the environment of the project presenting barriers to detecting early warning signals [U1], or simply not having a clear strategy [N5], or not being able to clarify and solve conflicts over goals or strategies [N5]. Finally, there were some barriers identified in the review process itself, including defined preassumptions in predefined formal assessments, preventing openness to early warning signs [N4], the opportunity to select areas for focus in gateway reports might make it possible to hide signs of problems [U1], and a belief that the project assessments would capture all problems preventing recognition of other early warning signs [N1]. We need to consider such barriers when trying to improve performance when implementing governance frameworks and project assessment methods in practice.

Assessments or “Gut Feeling” Approaches: Possibilities for Early Detection

Previously we saw that many early warning signs are of a less measurable nature and thus depend on more “gut feeling” approaches. The case studies gave an opportunity to look for signs of this phenomenon and identify further possibilities for early detection of early warning signs based on soft atmospheric or “feeling” issues. The main findings compared to typical “assessment-based” approaches are summarized in Exhibit 4, comparing the kinds of early warning signs which can be detected by each approach. The table is not complete in terms of covering every possible warning sign, but may help in understanding the significance of this division. By using formal assessments and looking for indications as exemplified to the left-hand side in the table, one is unlikely to detect the type of indications mentioned to the right-hand side of the table, unless very much aware of their potential as early warning signs.

Exhibit 4: Additional important early warning signs from case studies, sorted according to the way they are expected to be detected.

What Effect Did Complexity Have?

Complexity clearly gave rise to particular effects. [N4] noted that in a complex project everything is less well known and more interconnected, in particular reciprocal influences between the many elements [N2]. Such effects make analyzing causality in complex projects much more difficult. [U1] shows the difficulty in perceiving early warning signs arising from difficulty in comprehending the needs of a complex project. One particular lesson that could have provided an early warning sign in [U2] was that dyadic or bilateral contracts are not sufficient for an organizationally complex project. However, [N2] suggested that the advantage in a large complex project is the acceptance and awareness of the environment around the project. This, combined with the availability of resources (competence and capacity) of large, complex projects, helps success even in complex situations.

Why Were Early Warning Signs Not Acted Upon?

A large number of different EWS were detected, but usefulness is only realized if detected early warning signs are acted upon. Various case studies showed early warning signs being detected but not acted upon. In retrospect, it was clear that some things could have been differently, or some early warning signs were missed, implying a number of lessons. [U1] for example showed a project that was too large for the governance structure.

A number of projects showed the effects of politics, including:

• A “political” agenda and political pressure to implement some kind of a solution, and then later on overlooking warning signs [U1].
• Some projects may still be given high priority and be fast-tracked to approval if politically powerful interests stress the strategic need of these as being urgent [A1].
• Early warning signs which perhaps were not detected were the political influences themselves in a highly political domain [U2].
• Changing priorities and pressure to choose certain solutions despite previous decisions by the mother organization and the recommendations of the project team [N5].

Discussion and conclusions

Looking for Early Warning Signs

Initially in our study, we found that some of our key initial assumptions were confirmed: Current project assessment methods are established as a tool for the project owner to be reassured that the decisions about a project are based on sufficient fundamental facts and analysis. These assessments are generally performed as part of stage gate procedures and anchored in established governance frameworks. Most governments, corporations, and organizations do have such governance frameworks, formalized or not, on a high level. We also found proof that these assessments are limited is their ability to pick up early warning signs. They have an implicit focus of some issues (e.g., risks, progress, financial development) and thus turn “a blind eye” on other issues. Even when combining many different assessments on different levels and with a carefully fitted combination of focus areas, there might be issues and effects that slip through the formal assessments. What then can be done to improve this situation?

We looked into why established assessment methods fail to pick up early warning signs. The literature tells us that we are not good at seeing through complexity, uncertainty, and interpersonal effects. Our interviews and case studies added some details: Many attempts are reported at learning from previous projects, but these attempts are reportedly not very effective. Reasons indicated include lack of time to think about the critical issues, lack of time to prepare “lessons learned,” reluctance to “air dirty laundry,” an over-emphasis on the view of projects as unique, and the difficulty of learning from reports with insufficient contextual information. In the case studies, we find evidence of optimism bias leading people to overlook, or at least not act upon, early warning signs; indications of a lack of an external view (not external enough, not frequent enough), or too much trust placed in experienced people and consequently insufficient owner involvement. We find incidents of believing that assumptions will be stable permanently; we see motivational bias, which is hiding warning signs; attitudes which hold that “it is not my job” to shape the project and then passively overlook warnings; attitudes suggesting that a project is so urgent that there is no time to dwell on an identified problem (“failure is not an option”); actors insufficiently tough to tackle conflicts at their roots. Although some of the reasons for failing to pick up early warning signs are technical, the main challenges seem to be found within the minds of individuals. Additional issues include group thinking, blame culture, political pressure, and power effects. These are obviously good reasons to look at the way we assess projects again. What do we need to change in order to improve on identifying early warning signs?

Experience reported in the interviews and case studies showed that the assessment exercises (the process) and not the identified indicators were most important, because these allow crucial questions to be raised early. This is the first important conclusion; the exercise is more important than the result and the timing is decisive. An assessment raises awareness and allows for critical questions and discussions. Doing it early, on a stage where real options are still available, gives room for the assessments to represent a powerful tool. However, we see a tendency to introduce assessments too late. The arguments used to postpone are generally to wait for more facts, but the point is to ask critical questions about needs, objectives, and assumptions before the facts.

The case studies and interviews showed us that dialogue and organizational culture play a key part in detecting early warning signals. This confirms the need for “gut feeling” approaches that can capture signals which are not easily covered by more formal approaches. This confirms what Nikander and Eloranta (2001) and Whitty (2010) indicated in their work. This is the key to fixing the shortcomings of any formal assessment, or combinations of such. Where the formal methods are good at identifying EWS in the issues they are designed to look at, the informal “gut feeling” approaches are the possible way to look for signals without having a specific focus or issue in mind. Of course this is not an easy task. It cannot be learned by education or training from courses—it has to rely on the assessor's broad experience and deep understanding of both objectives and culture.

The literature shows that increasing complexity makes it more difficult to detect and interpret signals. There are clear indications that this is well known in the (public) governance frameworks, most of which are directed primarily towards large, complex projects (such projects are more critical, and they are the ones that have the resources to do thorough assessments). Interviewees similarly indicated that complexity was more important than size (although they can be related). The case studies show that complexity has some particular effects. Issues are less well known, and there are more interconnections (and more reciprocal influences, so it is difficult to deduce interactions from understanding the parts). Causality is less clear. It is difficult to comprehend needs and effects. Behavioral complexity makes patterns and positions difficult to understand. Complexity and dynamics in the environment are hard to foresee and respond to well.

Project management practices need to “fit” their context to provide value (Thomas & Mullaly, 2008), and contextual factors are important in defining complexity, so they also influence the appropriate choice of practices (Snowden & Boone, 2007); the interviews and the case studies support this. However, the governance frameworks and their guidelines suggest similar approaches and assessments across sectors and project types. One of the most interesting observations in our material was this mismatch between actual practice and the previous two conclusions. There seems to be a tendency to increase the number and frequency of formal assessments with the complexity of the projects. Our findings indicate that the real need in more complex situations may be more “gut feel” based approaches.

The consequence is changing governance and management methodology: by implementing formal assessments, the dependence on “gut feel” seems to be reduced as does the ability to use “gut feel” to detect early warning signs. On the positive side, we find that large, complex projects do have awareness of the surroundings, as well as more time and resources to influence their situation. On the negative side, our interviews and cases indicate that there are important obstacles in interpreting and identifying clear signals in complex situations—which maybe suggests that formal assessments, which are usually dominated by analytical approaches, may actually be inappropriate (not sufficient) in complex projects. We saw, in some cases, examples of how analytical approaches had been useful in avoiding potential problems, but even more examples of these approaches not succeeding in doing so. The interviews gave indications that there is a mix of “hard issues” (technical and more measurable) and “soft issues” (people issues (e.g., attitudes and values) and harder to measure) among the early warning signs. In general, the interviewees advised: use an outsider's view, be anchored on a high level in the permanent organization, and look for inconsistencies to detect lack of trust, signs of inappropriate culture, lack of clarity in thought, and misalignment between qualitative and quantitative analysis. They confirmed that “knock-on effects” occur frequently. They recommended solving issues immediately and thoroughly.

What Early Warning Signs to Look for

Which are the most important early warning signs to look for in the different contexts, and why? The guidelines gave us a list of 30 important potential early warning signs—which could have been much longer, given the rich basis of the checklists—but with no indication of relative importance. The list is context independent, except that the basis on which it is developed came from large, complex projects. We found the expected duality between “hard” and “soft” issues expressed in the early warning signs, naturally “hard” issues being identified typically through formal assessments and “soft” issues being identified through gut feeling-based-assessment.

Our interviews can be compared to Meier (2008), who investigated US Federal Intelligence and Defense Agencies projects and found EWS manifested by: overzealous advocacy; immature technology; lack of corporate technology roadmaps; requirements instability; ineffective acquisition strategy and contractual practices; unrealistic program baselines; inadequate systems engineering; and inexperienced workforce and high turnover. Meier's factors are more specific, while our results are formulated on a higher level; we are looking for the important early warning signs—Meier for the most frequent ones; our interviewees represents a wider set of project types and organizations in three countries, whereas Meier's sample is one specific type of context in one country. Factors such as leadership and culture are not present in Meier's results, which have a strong focus on contract and technical (“hard”) issues. This may be due to the cultural context of Meier's data; our results do not have contextual elements included (with the exception of “location decisions” indicating physical infrastructure). However, context really does matter, and that it is especially important to understand the cultural conditions around and in the project to be able to pick up the earliest warning signs.

Kappelman et al. (2006) gives a good list of early warning signs, and one that explicitly ranks them according to importance; these are for IT-projects, but the warning signs seem relevant to most other types of projects as well. Again comparing these to our interviews gives Exhibit 5.

Exhibit 5: Comparison of interview results with Kappelman et al. (2006).

Both point out important aspects of documentation, the business case/relevance of solution, leadership issues, and competence of the team. Some aspects in the two columns are related, such as “risks” and “change control” or “quality of information” and “lack of requirement/criteria”; we have location decisions and their consequences, and Kappelman et al. (2006) have stakeholders and their involvement and communications. Our interviewees perhaps have an external, project owner focus, and Kappelman et al. (2006) have a more internal, project management focus (e.g., we mention governance guidelines and culture; they mention ineffective planning and change control management, etc).

Exhibit 6 shows a structured summary of all our empirically based early warning signs from interviews and case studies. Formulations with the same or similar meaning are removed although some occur in more than one phase. We can compare this list to the literature although of course our results do not cover all possible early warning signs (e.g., the specific technical and contract-related warning signs in Meier (2008)). Firstly, the division between assessment based and gut-feel-based early warning signs is parallel to the division line between authors such as Jaafari (2007) and Oakes (2008), who focus (only) on assessment based indicators, and researchers such as Nikander and Eloranta (2001) and Whitty (2010) who hold “gut feelings” and non-verbal information, individual's feelings, differences in culture, and also miscommunication as main sources of early warning signs. We strongly hold that in order to see through complexity one will need both kinds of approaches. Secondly, we have included signs at the time of project set-up, in the early stages, and during project execution.

Exhibit 6. Early warning signs extracted from our empirical studies.

We tend to look at the projects from the owner's perspective and have a strong governance flavour and external focus. This is especially evident in the project set-up and early stages columns; the project execution column, on the other hand, has more project management issues. Comparing our tables to the literature reveals that our empirical results have more in common with the early warning signs extracted from guidelines of governance frameworks than other research results. This is especially true for the assessment-based indicators in the first two columns; the more project execution related results compare quite well to Kappelman et al.'s (2006) results. We have also identified a few early warning signs that we have not found in the other quoted literature: people in acting positions without authority (may be similar to “weak project manager”), project team over-relying on others to fix problems, team members working too much or too little, contractor unfamiliar with domain, lack of commitment to make decisions, parties making reservations, changes in positions over time, unwillingness to conclude, considering needs as unreal, lack of trust, the way answers are given to critical questions, and continually unfulfilled promises

Why Early Warning Signs Are Not Acted Upon

Early warning signs are possible to identify, as we have seen in several case studies. However, we have found little evidence that project managers have found ways to exploit these to identify and deflect future problems. This leads to the next issue: why then are they not acted upon? Fundamentally, it is hard to do when there is no time to think ahead and question assumptions, so time pressure is one reason. This explains why there is little motivation for spending time extracting and using lessons learned. Another organizational reason for this is the mismatch in incentives between the organization and the individuals. The individuals bring the experience with them to the next project and do not see the need for securing the ability of the organization to learn. Earlier we also found a number of reasons why early warning signs are overlooked, and many of them are also reasons why identified signals are not acted upon. Among the important ones are: the tendency for optimism; the trust in the projects ability to run faster, and fix the problems and it will be OK in the end. A number of projects showed the effects of politics, including a “political” agenda and political pressure to implement a given solution and powerful interests stress the strategic need of the solution as being urgent. This indicates that the responsibility to actually make effective actions possible may not lie with the project managers, but with the project owners.

Contribution and Further Research

To summarize our study:

• Systematically links stage gate models, project assessments, and early warning signs. We perhaps now have more elements for a model that may be used to explain and/or detect early warning signs.
• Contributes a richer background for claiming that every project is unique and has to be analyzed on its own to detect early warning signs.
• Ssupports the use, under given circumstances, of some simpler rules of thumb.
• Indicates the usefulness of the “hard” indicators in the moderately complex projects.
• Indicates the “soft” nature of many early project problems and corresponding warning signs (e.g., political, communication, leadership, and culture).
• Shows “gut feel” indicators are important, particularly in complex situations and that some can be characterized or systematized; this may serve as an input to improve governance frameworks and guidelines.
• Shows the difficulties with organizational culture in identifying such “gut feel” indicators.

We have also identified some ideas for further research:

• We studied a variety of governance frameworks, but what really happens in practical terms within these arrangements would be a useful study (some of this done by the Concept research program for the Norwegian Ministry of Finance and in the UK by the Office of Government Commerce and the National Audit Office).
• One consequence of the industry/project-type specific nature of the findings is that there is a need for such specific studies to consider which early warning signs are the most important in each industry/project type.
• There is a need for more empirical research on the consequence of the degree of complexity in the situation. There are also indications in our case studies that current assessments methods are not good enough when the situation is very complex.
• The problem of identifying and dealing with unknown unknowns remains. More case studies of such situations could provide realistic scenarios to research concerning possible ways of dealing with these.
• We did not make much progress towards identifying the “knock-on effects” before they happen; this is important in answering the question of how to see through the complexity. There is need for more empirical studies or action research.
• These results are for three countries, one Scandinavian and two Anglo-Saxon, and more research is needed in other cultures.

Acknowledgements

This research would not be possible without financial support from the Concept Research Programme, Project Management Institute, and Norwegian Centre for Project Management.