How to deliver successful projects with increasing socioeconomic and political complexity?

Introduction

There are compelling reasons for learning to solve the right problems. First, strategies for solving tame problems differ qualitatively from strategies appropriate for ‘Messes’, messes are puzzles; rather than solving them, we resolve their complexities (Ackoff 1970). In turn, solving and resolving both differ qualitatively from strategies for dissolving the barriers to consensus implicit in ‘Wicked’ problems (Ackoff 1974). A second and more compelling reason is not so much that solving the wrong problems fails to solve the right problems. Rather, the greater danger is that by solving the wrong problems, we unwittingly undermine what it takes for us to solve the right problems. The danger is not so much that we fail to build our bridges across the right rivers. Rather, the greater danger is that we destroy the materials we need to build our bridges across those rivers. A third reason for learning to solve the right problems is that powerful technologies are being rapidly developed which give every indication of generating messes, if not wicked problems and wicked messes.

Tame Problems

Do not confuse tame with simple, we solve tame problems through analytical methods, breaking things down into parts, fixing components and assessing the probability of known sequences of failures leading to an accident. We organise ourselves to solve tame problems through specialisation. Culturally, tame problems enjoy consensus: everybody pretty well agrees why something needs to be done and the right way to go about doing it. There are countless examples of tame problems, solving them has been the great forte of science for several hundred years. Due in large part to such successes, they remain the ideal for many social scientists as well as managers and administrators (Exhibit 1).

Exhibit 1: Traditional wisdom for solving complex problems. The “waterfall”

Messes

However, things have become more complicated. We are increasingly faced with problems of organised complexity, clusters of interrelated or interdependent problems, or systems of problems. Ackoff labelled this system a ‘mess’. Problems, which cannot be solved in relative isolation from one another form messes. We sort out messes through systems methods, through focusing on processes and through interdisciplinary approaches. Rather than simply breaking things down into parts and fixing components, we examine patterns of interactions among parts. We organise ourselves to sort out messes through such things as cross-functional groups, redundancy, and so-called ‘learning organisations’ (Senge 1990, 8). Many examples illustrate the concept of messes. Simply building more motorways doesn't solve vehicle congestion. A risk in mistaking a mess for a tame problem is that it becomes even more difficult to deal with the evolving mess. Karl Weick (1979, 147) noted that “Managerial problems persist because managers continue to believe that there are such things as unilateral causation, independent and dependent variables, origins, and terminations.”

In his book Normal Accidents (1999, 78) Charles Perrow details some of the problems inherent in messes. Firstly ‘Interactive complexity’ : this is the measure of the degree to which we cannot foresee all the ways things can go wrong. This may be because there are simply too many interactions to keep track of. More likely, it is because our various theories are simply not up to the task of modelling socio-technical interactions and ‘coupling’; this is a measure of the degree to which we cannot stop an impending disaster once it starts. This may be because we don't have enough time, because it is physically impossible, or because we don't know how. The greater the degree of interactive complexity the less our capacity to prevent surprises and the greater the degree of coupling the less our capacity to cure surprises. Therefore, the greater the degree of interactive complexity and coupling, the greater the likelihood that a system is an accident waiting to happen what he terms a ‘normal accident’(5). In such systems, ‘operator errors’ merely serve as triggers. Strategies for dealing with messes are therefore quite different from those appropriate for tame problems. Strategies logically follow from the ways problems are conceptualised. Thus, increasing our capacity to prevent unanticipated interactions among components entails simplifying systems (KISS); increasing our capacity to cure them entails de-coupling major components (e.g. build in longer times-to-respond).

Wicked Problems

In short, strategies for dealing with messes are fine as long as most of us share an overriding social theory or overriding social ethic. If we don't, we face ‘wickedness’. Wicked problems are termed as ‘divergent’ as opposed to ‘convergent’ problems. A convergent problem promises a solution. The more it is studied, the more various answers sooner or later converge. Tame problems are convergent by definition. Messes are convergent if we agree on what overlaps, on appropriate strategies, and on the kind of ‘climate’ we wish to maintain. A divergent problem does not promise a solution. The more it is studied, the more people of integrity and intellect inevitably come to different solutions. As with messes, there are very real dangers in ‘solving the wrong problem’. Mistaking or misrepresenting wicked problems for messes, let alone tame problems, almost inevitably leads one to conclude that those with different answers lack integrity, intellect, or both. The great danger is that such conclusions undermine trust, and trust is a fundamental strategy for collectively coping with wicked problems. If wicked problems are becoming more common in our modern era, and there is compelling evidence they are, we face a strategic choice. We can continue to misrepresent them as messes or tame problems, hoping they will not degenerate. On the other hand, we can acknowledge wicked problems for what they are and try to stabilise them as ‘conditions’. This is not going to be easy because wicked problems offend our sense of logic and our common beliefs even more than messes. In our modern times, it is pretty hard to accept a problem has no solution; this seems tantamount to giving up. Given that many people care about or have something at stake in how the problem is resolved, the process of solving a wicked problem is fundamentally social and solving a wicked problem is fundamentally a social process. Most wicked problems involve lots of stakeholders.

Wicked Messes

Much research has been done on behavioural complexity and dynamic complexity. What confuses real decision-making is that the two co-exist and interact in what we call “wicked messes.” (Senge & Roth 1996, 95)

Exhibit 2: Boston Matrix showing the 4 types of problems

The fact that problems cannot be solved in isolation from one another makes it even more difficult to deal with people's differing assumptions and values: people who think differently must learn about and create a common reality, one which none of them initially understand adequately. Systems of interlinked problems interact with the misunderstandings, divergent assumptions, and polarised beliefs of different groups of people. Improving communication and trust among different camps is not enough; people are still likely to focus on symptoms rather than deeper causes and pursue low-leverage changes. Conversely, even if deeper understanding of the systemic forces at play is achieved, such understanding will be viewed with suspicion by the different, competing interests and mental models.

Solutions to Wicked Messes

The main thrust to the resolution of these types of problems is stakeholder participation and ‘satisficing’ (Simon 1956, 77). This is because wicked problem resolving produces no single optimum solution but has many alternative satisfactory solutions. Therefore the production of that solution must be ‘boxed’ or ‘bounded’ either by time or financial constraints to avoid it going on ad infinitum.

So how do we propose to deal with wicked messes? Wicked problems are synonymous with what Geertz (1973, 219) terms “a loss of orientation” or what Rittel and Webber term the absence of an “overriding social theory or an overriding social ethic.” Thus, wicked messes are evidenced by the ideological controversies that result when the boundaries of messes expand to include socio-political and moral-spiritual issues. So-called empirical studies and the social sciences are necessarily shot through with implicit and explicit value assumptions and ideological considerations. ‘Wickedness’ occurs when people confer immutability on value assumptions and ideological considerations. Thus, the strategic issue is whether we choose to allow wicked messes to degenerate into chaos, whether we choose to stabilise them as ‘conditions’, or, more radically, whether we choose to try to dissolve them together.

Leadership versus management

How can this matrix aid us with in project management? For the purposes of this paper it is key to draw a distinction between management and leadership. Kotter (1990, 103) states that leadership is different from management. Management, he goes on to say, is about coping with complexity, its practices and procedures are largely a response to the emergence of large organisations and complex projects; leadership, by contrast, is about coping with change. These different functions, coping with complexity and coping with change, shape the characteristic activities of management and leadership. Each system of action involves deciding what needs to be done, creating networks of people and relationships that can accomplish an agenda, and then trying to ensure that those people actually do the job. But each accomplishes these three tasks in different ways. Therefore how can we choose when we require the skills of leadership to dominate against those of management on a project to obtain a successful outcome? Here I propose the use of our matrix. If we now revisit exhibit 2 and look at the characteristics required to address these issues, we can see those required to achieve change are predominantly behavioural based requiring needs and emotions to be satisfied indicating a need for a degree of influence rather than power to ensure their resolution based on the development of trust and relationships, indicating a high degree of behavioural complexity. This allows us to segment the matrix into two where leadership qualities dominate solutions in the upper half and management qualities the lower. (Exhibit 3)

Exhibit 3 : Boston Matrix with suggested leadership and management skills overlaid

The correlation between risk, tame problems, messes and wickedness

In the past there has been a considerable difference of opinion between the use and legitimacy at various times of qualitative and quantitative risk assessment (Oldfield 2001). I propose the following hypothesis to why this may have occurred. If we accept the preceding classification of types of problems as shown in Exhibit 3 and then analyse how risk analysis may be applied to help us with each type of problem. We can see that as we move further to the right of the matrix there appears to be an increased reliance on the use of systems thinking with the emphasis on the interconnectivity of events. As we move further towards the top of the matrix there seems to be an increased reliance on social science type solutions. (Where people do not necessarily behave as ‘soft systems’ but their behaviour is dominated by belief systems based on emotion and feelings). If we then apply our two types of risk assessment (quantitative and qualitative) and overlay them on the matrix we produce the diagram as shown in Exhibit 4.

Exhibit 4: Boston Matrix with suggested risk assessment processes overlaid

We can observe that, for the bottom sectors where scientific based solutions are dominant, quantitative risk data is reliable in the assessment of risk. Our assumption here is that there is an optimum solution and therefore an underlying number that we will converge towards. Thus by the generation of more and more data we will move closer to the underlying ‘truth’. However as we move up towards the behavioural sector there is no underlying single solution it will now be dependent upon the feelings and ideologies of the people concerned in the assessment. Here by the use of quantitative data to inform facilitated workshops, qualitative results will ‘reveal’ those underlying beleifs which will lead to ‘satisfactory’ solutions. Therefore it is critically important to involve all stakeholders who can influence the success of the project and to attempt to limit your boundaries to the case in hand as otherwise the group will grow beyond the limits of acheiving a workable solution.

Case Study: Terminal 5 Project (Heathrow)

Terminal Five as a Tame Problem

The solution that Terminal 5 was expected to provide initially was viewed as the need to build a building to process 20 million passengers per annum. This determined the project predominantly as a terminal building with some other sub processes surrounding it. The project then developed around the terminal building with smaller project teams evolving their supporting areas in isolation. This meant projects were optimised with out little concern for the impact on the other teams. The result was a rather grand terminal building with all sub units equally optimised. This produced a cost envelope outside of the budget. For risk management a risk register was developed in a single workshop during the early stages by the project team containing predominantly risks of a technical nature. This was developed into a project document, filed and was not revisited during that phase.

Terminal Five as a Mess

As the project progressed it became apparent that it did not comprise a set of separate, independent problems but consisted of many interacting complex systems and subsystems this incompatibility was highlighted by a baseline review where the project was ‘brought together’ to see how it would operate as a terminal and whilst the individual units were optimised the overall product was sub optimal. This led to the problem being approached as a ‘mess’. Here workshops were developed to discover which teams would be required to inter relate and where these major interactions would take place. This review consequently discovered many complex interactions between teams that occasionally surprised the team members who had been working in isolation. This produced the first changes in behaviour with the teams providing short duration programmes where they would come together to test their solutions, review them then depart and try and optimise their part of the problem before returning to test their revised result against the overall solution. This type of rapid prototyping helped develop a much more integrated solution. However there were still some problems, which appeared to go round in circles without seeming to get any closer to an agreed solution.

Terminal Five as a Wicked Problem

Here we introduced the concept of wickedness to the project, which appeared to explain our experiences in designing the ‘Ultimate Passenger Interchange’.

1. Wicked problems have no definitive formulation. Formulating the problem and the solution is essentially the same task. Each attempt at creating a solution changes your understanding of the problem. On Terminal 5 as each component part was optimised it changed its relationship with the overall solution.
2. Wicked problems have no stopping rule. Since you can't define the problem, it's difficult to tell when it's resolved. The problem-solving process ends when resources are depleted, stakeholders lose interest or political realities change. We continually interchanged check in desks, staff facilities and retail space in the terminal each having a ‘slight’ advantage from a certain viewpoint.
3. Solutions to wicked problems are not true or false, but ‘good’ or ‘bad’. Since there are no unambiguous criteria for deciding if the problem is resolved, getting all stakeholders to agree that a resolution is ‘good enough’ can be a challenge. Here the conflict between retail and airport operations was most apparent based on their subjective viewpoints.
4. There is no immediate or ultimate test of a solution to a wicked problem. Solutions to such problems generate waves of consequences, and it's impossible to know how these waves will eventually play out. This has been shown by some commitments that were made during the public inquiry phase only now being understood in the design stage.
5. Wicked problems don't have a well-described set of potential solutions. Various stakeholders have differing views of acceptable solutions. It's a matter of judgment as to when enough potential solutions have emerged and which should be pursued. Every implemented solution to a wicked problem has consequences. Over the long period of the project design phase (7 years) the stakeholders involved have changed with a subsequent change to the overall ‘acceptable’ solution.
6. Each wicked problem is essentially unique. There are no ‘classes’ of solutions that can be applied to a specific case. Part of the art of dealing with wicked problems is the art of not knowing too early what type of solution to apply. (Rittel and Webber, 1973, 159)
7. The causes of a wicked problem can be explained in numerous ways. There are many stakeholders who will have various and changing ideas about what might be a problem, what might be causing it and how to resolve it. At workshops this was abundantly apparent and much time was spent ‘discussing’ why an individual's particular solution was considered the best approach.

Introspection also revealed that this facility would incorporate passengers for whom their travel ‘experience’ would define whether T5 would be viewed as a success. Complications were added to by Terminal 5 having many stakeholders (local council, environmental pressure groups, highways agency, rail operating companies etc) and being dominated by two major (and virtually equal) stakeholders; BAA and British Airways who viewed success differently and had different (sometimes completely opposed) viewpoints and strategic goals, this placed us squarely in the region of wickedness. To help with this we developed facilitated workshops to deal with the social aspects of the problems. We also introduced the concept of wickedness to stakeholders and why consensus and ‘satisfactory’ solutions were crucial to the success of the project. Here the use of facilitated workshops led us to a better understanding of the risks that may obstruct the project outside of the purely technical arena that in the beginning appeared to be the dominating factors of uncertainty on the project.

Terminal Five as a Wicked Mess

Much research has been done on behavioural complexity and dynamic complexity. What confuses real decision-making is that the two co-exist and interact in what we call “wicked messes”. The concept of the passenger as seen by the two major stakeholders illustrated this perfectly. You would expect that the airport and airliner would be a perfect symbiotic relationship. However in BAA's case it generates a large percentage of revenue from retail therefore it seeks to maximise the dwell time in the terminal, whilst BA, in providing their service, (who are now pursuing the business section of the industry) seek to reduce the time of the passenger in the terminal and any dwell time to be spent ‘at leisure’ in their lounges. Therefore it is not their primary spend (travel) where the conflict arises but on their concessionary spend; duty free purchases, meals and other retail. How therefore did we deal with these tensions during the T5 project. From the risk perspective we introduced the concept of risk champions and facilitated risk workshops (Hancock 2000). With these methods the stakeholders were allowed to seek different solutions as they explored divergent and convergent aspects of their views. At this point, they would be engaging in dialogue. Rather than arguing their respective positions and trying to persuade the other their solution was the correct one, they would start rethinking together, some of their basic assumptions. This proved to be problematic at first as the members felt vulnerable and trust was lacking between the various contractors but as time progressed and they realised they were both seeking satisfactory rather than optimal solutions they continued to talk together to explore solutions. At some point during this process they would realise that they shared common ground. This essentially meant both realised that their differences were less significant and profound than what they shared in common, and that this common sense began to form the basis of the integrated T5 solution. The stakeholders involved have not yet gotten to the point where they all trust one another but they have realised there is a need to work towards a ‘common good’. The strategic principles for establishing trust is grounded in the realisation that what we share in common are far more significant and profound than our differences. In sum, these strategic principles are essential to sorting out wicked messes together. More significant, in our increasingly complex and interdependent times, these principles guide us to live in ways that help us dissolve what is problematic.

Resolving wicked messes

Based on experience the author suggests there are predominantly three ways of dealing with wicked messes, based upon the distribution of social power within the group.

Social Power

Cartwright (1959, 150) states that “power is inevitably a part of the accepted phenomena of social psychology” however the task of documenting what power actually is, or how we define it, is made difficult because of the ambiguity concerning its boundaries. For ease we will use the bases of social power as illustrated by French & Raven (1959). Here they define power in terms of influence and influence in terms of psychological change. Psychological change for them is defined as the “alteration of the state of some system over time”. Social influence and power is this case is limited to the change effected by an individual (P) over another individual or group of individuals (O).

The basis of Power

By the basis of power we mean the relationship between O and P, which is the source of that power. French maintains that there are five bases for O's power.

1. Reward Power - P's perception that O has the ability to mediate rewards for him
2. Coercive Power - P's perception that O has the ability to mediate punishments for him
3. Legitimate Power - the perception by P that O has legitimate right to prescribe behaviour for him
4. Referent Power - P's identification with O and
5. Expert Power - the perception that O has some special knowledge or expertise

For all types of power the stronger the basis of power the greater the power. Some negative aspects are produced by coercive power in that it results in an increased resistance towards the individual exerting that power and conversely reward power produces a reduction in that individual's resistance. This can be offset by the fact that the more legitimate the coercive power the less it tends to produce resistance. Reward power will also increase referent power over time as it increases the attraction of P towards O. Whilst most of these powers are broad ranging expert power is limited to specific areas and use of expert power outside of these areas can undermine and reduce the effect of that power.

Identification of stakeholders

We begin our identification with Freeman's (1984, 46) definition of what constitutes a stakeholder “any group or individual who can affect or is affected by the achievement of an organizations objectives”. This was further subdivided into classes based on (Mitchell 1997):

1. The stakeholders power to influence the firm
2. The legitimacy of the stakeholders relationship with that firm and
3. The urgency of the stakeholders claim on the firm

This can be distilled into stakeholder ‘salience’ to explain how managers prioritise their stakeholder relationships. Where salience is defined as “the importance the individual attaches to supporting or opposing the objective relative to all other decisions with which the individual is concerned”. (Coplin 1976, 15). Therefore salience is considered to be high where all 3 of the stakeholder attributes are perceived by individuals to be present.

Strategies for dealing with Wicked Messes

1. If power is dispersed among a small number of stakeholders then authoritative strategies can be used. These are taming strategies. They diminish the level of conflict inherent in wicked messes by putting problem solving into the hands of a few stakeholders who have the authority to provide a solution, the so-called ‘experts’. The advantages are that reducing the stakeholders reduces the complexity. The disadvantages are that what is achieved may please none of the stakeholders involved and ‘experts’ can be wrong.
2. If power is widely dispersed, but cooperation appears impossible in the relevant timescale, then a process of ‘muddling through’ is usually applied. This is where the project deals with problems as they arise on a whoever ‘shouts the loudest’ or makes the ‘strongest argument’ basis. The project meanders through the process until it ‘arrives’ at a solution. Advantages are typically short lived and the disadvantages are that the problem is never ‘fully’ resolved as the solution is constantly changing. Many political problems typically fall into this category.
3. If power is widely dispersed and time and consensus is not prohibitive then collaborative strategies can be used. These have at their core a win-win solution and provide ‘satisfactory’ solutions (Simon 1956, 77). Alliances, partnerships and joint ventures are all variations on the collaborative theme. Advantages are numerous and have been catalogued in many other papers. Disadvantages are that they take a great deal of effort to manage and achieve synergy. Dialogue can turn into debate and debate into procrastination with little to show for it and at the same time transaction costs can increase whilst the dialogue progresses.

Collaborative strategies are the most difficult to implement but the most rewarding. One way of achieving alignment of stakeholders is through the use of ‘customer experience’ i.e. what type of experience are you seeking for the end users. This focus can instil in the stakeholders a ‘motivating proposition’ for the project. By mapping stakeholders within the T5 project and looking for ‘common ground’ the concept of the “Worlds most refreshing interchange” was developed. This has given substance to what is acceptable within the project but more importantly what is unacceptable.

Conclusions

The first step in solving problems is to recognise the type you are facing. To help the reader to achieve this I have attempted to classify problems into four main areas (Tame, Messes, Wicked Problems and Wicked Messes). This has proved useful on the T5 project. We have also used the matrix to try to understand the behaviours required to deliver a successful project using managers or leaders. I have attempted to overlay the concepts of risk management to use the right risk management techniques with the type of problem. Having recognised you have a wicked mess the paper has attempted to come up with some coping strategies for dealing with them, which hopefully many of you will recognise and the use of customer experience for alignment of stakeholders. Many situations on major projects are complex both behaviourally and systemically. These situations need to be identified and recognised and then for suitable solutions to be progressed using the correct tools. The solution to wicked messes lies in the people, by the use of discussion and forums for exploring the problems and their understanding of the limitations of the solution. Accurate reporting of ‘near misses’ and the use of workshops to identify root causes can aid this process. However, for this to be achieved requires an open culture, where an attitude of ‘no-blame' exists. It also requires the pursuit of designs that limit system coupling and complexity where possible.