Dancing in the kaleidoscope
the challenge of leading complex projects
Complex projects, like kaleidoscopes, are multifaceted. They are characterized by emergence and inter-connectedness, cannot be isolated from their environment, and are highly subject to the influence and reactions of people with different agendas, values, and responses. Such projects need good leadership from people who are self-aware, collaborative, comfortable with ambiguity, able to think systemically, and take a kaleidoscopic view, while acknowledging the validity and value of different perspectives. This presentation will introduce a variety of tools and approaches that can be used to provide understanding of the many facets and perspectives of complex projects and help leaders and teams “know what to do when they don't know what to do.”
Complexity in projects has become a very popular theme in recent times, and interest in complexity is not restricted to the world of projects. It is a hot topic in all fields of endeavour, and particularly in the natural and social sciences and in general management. In all fields it tends to be associated with a move away from belief in control and an expectation that everything is ultimately amenable to scientific explanation toward recognition that we can't necessarily understand, predict, or control our environment. For some, this is particularly challenging, as traditional approaches to project management embody assumptions that optimum solutions and control are possible and that endeavours can be simplified by reducing them to their component parts. While this may never have been fully realistic or achievable, it has become increasingly challenging in the face of changing social values and individual empowerment.
To meet the challenge of dancing in the kaleidoscope, we need to understand the multifaceted, systemic nature of complexity in projects, and the challenges this represents for leadership. This paper begins with a brief review of what we mean by complexity in projects and the challenges it represents. Engaging stakeholders in understanding and making sense of complexity in projects is one of these challenges, and a number of frameworks that can be used for this purpose are also introduced. Finally, implications for leadership are discussed.
Understanding Complexity in Projects
Complexity is a term used often to denote something that is perceived as difficult or challenging either to do or to get our head around. We all have our own ideas of what it means, and the meaning may vary according to context. A useful distinction can be made between descriptive complexity dealing with characteristics such as technological and organizational complexity (Baccarini, 1996) or structural complexity and uncertainty (Williams, 2002) and perceived complexity (Vidal & Marle, 2008), which is subjective and very much in the eye of the beholder. It is also useful to distinguish between projects that are complicated and those that are complex. Snowdon (2002) provides an excellent illustration of this distinction:
An aircraft is a complicated system; all of its thousands of components are knowable, definable and capable of being catalogued as are all of the relationships between those components. … Cause and effect can be separated and by understanding their linkages we can control outcomes… . Consider what happens in an organisation when a rumour of reorganisation surfaces: the complex human system starts to mutate and change in unknowable ways; new patterns form in anticipation of the event. On the other hand, if you walk up to an aircraft with a box of tools in your hand, nothing changes. (p.105)
It is the human factor, or the involvement and influence of people, combined with context that introduces complexity to projects. It is people, also, who perceive different levels of difficulty and therefore complexity based on their own level of experience and confidence. The role of people in the complexity of projects can be explained by thinking of projects as complex adaptive systems.
Projects as Complex Adaptive Systems
Complex adaptive systems (CAS) are dynamic networks of multiple components or agents that interact in non-linear ways, self-organizing, evolving, learning, and adapting to changing internal and external environments. The behaviour of the system is a result of its history and numerous decisions, actions, and reactions between the individual agents. There are positive and negative feedback loops within the system, but each element or agent is ignorant of the behaviour of the system as a whole, responding primarily to information available locally (Cilliers, 2000). CAS are generally considered to be open systems due to interaction with their environment, and there is difficulty in defining the borders of the system, which will vary according to the purpose and the observer. They are unpredictable and have emergent properties that result from interactions within the system and with the environment. Like the kaleidoscope, the components of the system shift and change, forming new patterns.
If we think of these interacting components as people, projects can be seen as complex adaptive systems comprising networks of people (agents), including the project team and other stakeholders, influenced by their own worldviews and histories, acting and reacting to one another and to their environment, sharing information formally and informally. Thinking of projects in this way is useful because it allows us to draw upon the characteristics of complex adaptive systems as a framework for understanding the complexity in projects and the challenges this represents.
Approaches to Understanding Complexity in Projects
Dimensions of Complexity
Although people are central to the complexity of projects, and all projects involve people, the sensitivity of the project to human interaction, and therefore the level of complexity, will vary. Many organizations categorize the projects in their portfolios according to their degree of complexity, but complexity is not a single construct, so such categorization systems, in practice, are based on multiple attributes such as scope, clarity of goals, and objectives, and level of ambiguity and uncertainty (Crawford, Hobbs, & Turner, 2005). Exhibit 1 presents dimensions of complexity that can be used to guide understanding of the nature of complexity in projects. A project that exhibits all the characteristics shown on the left hand side of the diagram could be considered more complicated than complex, more a closed than an open, complex, and adaptive system.
Exhibit 1 – Dimensions of Complexity (adapted from Crawford, Sankaran, & Butler, 2005)
Paradoxically, projects can be both complicated and complex. In other words, parts of the project may be best described by terms and concepts characteristic of complicated projects, while other parts of a project may be best described by the terms representative of complexity. With advances in technology, it is reasonable to propose that projects have become more technically complicated with time, but if we define complexity as a function of the involvement of people, then it is also reasonable to propose that this has always been a condition and may be more significant in particular types of projects or under particular conditions.
With tangible end products such as buildings, equipment, and machinery, projects can be complicated. The end products can be produced as physical models and prototypes providing a basis for clarity in terms of what will be delivered. Where the end products of the project are intangible, such as in information systems and organizational and cultural change, clarity and agreement about the end product are far harder to achieve. They do not lend themselves to physical modelling or prototyping, and people are likely to have different interpretations and expectations of what will be produced. Judgement concerning project success is far more difficult to assess in projects with intangible products. Associated with this, complicated projects lend themselves to being well-defined, while complex projects are often ill-defined.
Complicated projects tend to have hard, clear boundaries that well define what is in scope and what is not in scope, with the boundaries often defined by contracts. There is very little exchange between the project and its environment, and any exchange that does occur is well controlled. At the other end of the spectrum are projects where the boundaries are soft and permeable. An example is an organizational change project where it is difficult to identify where the project starts and ends and where it is possible to move or redefine the project boundaries. Staff internal to an organization generally undertakes such projects and is not isolated from their environment by contractual structures. These personnel are often shared with other projects as well as functional or line positions, further blurring the boundaries of the projects. Consequences of this include difficulties in managing project scope, reporting, and accurately determining project costs.
Complicated projects, with their tangible end products and hard, clear boundaries, tend to be unambiguous and characterized by a focus on goal achievement. It is usually relatively easy to articulate what the project is and what it is intended to deliver with clear well-defined goals, and as Winter and Smith (2005) suggest, the goal is “given” at the start. Complex projects are generally ambiguous, in that it can be difficult to describe the project, its purpose, and what it is intended to deliver, and to do so in a manner that is always interpreted consistently. There are often multiple purposes and goals that are initially ill-defined and emerge as a result of negotiation and consensus building throughout the project.
In complicated projects it is assumed that there is one best solution generally developed by technical experts. Complex projects rely upon multiple world views and perspectives, with solutions developed through negotiation and debate between multiple stakeholders. The path to the known and agreed best solutions is managed, while in complex projects the process of arriving at possible solutions is facilitated. Consistent with this theme, it is possible to plan in advance the strategy for complicated projects, while the strategy for complex projects tends to be emergent.
Clear boundaries that provide a degree of isolation from environmental influences allow for the use of intricate tools and techniques of project management and are suited to complicated but stable projects. Such tools and techniques include network planning, risk analysis techniques such as Monte Carlo analysis, and management utilizing computerized resource allocation and task coordination. As the boundaries of the project become more permeable and the descriptors in Exhibit 1 that are associated with complexity become applicable to the project, the use of many of the tools and techniques of traditional project management becomes more difficult, less reliable, and further removed from reality.
When the goals, methods, and end products of projects are known, a primary responsibility in management, is the reduction of residual uncertainty. In complex projects the focus is on ambiguity reduction, or on progressively negotiating and achieving clarity, agreement, or accommodation in terms of goals, methods, and outcomes.
Using this set of dimensions, or the hard and soft dimensions suggested by Crawford and Pollack (2004), to generate discussion can lead stakeholders toward an appreciation of the level of complexity in projects, and ways to deal with it effectively.
Goals and Methods Matrix
Another useful approach for understanding and communicating the complexity of projects is the Goals and Methods Matrix attributable to Turner and Cochrane (1993), with colorful titles from Eddie Obeng (1994). When read in conjunction with the attributes presented in Exhibit 1, Type 1 (painting by numbers) projects (Exhibit 2) where the goals and methods for achieving them are well known can be seen as typical of the complicated end of the spectrum, and Type 4 (walking in the fog) as having characteristics of complexity. Types 1, 2, and 3 projects may, however, have begun as Type 4, or walking in the fog projects. They may even be sub-projects of projects or programs, whereby highly complex endeavours are progressively elaborated and achieved. The whole picture may not be known at the start and, as a complex adaptive system, the overall shape and direction may change, but progress is made iteratively through smaller, well-defined projects with clear goals. Where either the goals or the methods of these sub-projects are unknown or unclear, a degree of complexity may remain. This framework can be extremely useful in helping all stakeholders understand the differing levels of complexity throughout a project or program, and to set expectations and apply management methods appropriately.
Exhibit 2 – Goals and Methods Matrix (Sources: Turner & Cochrane, 1993; Obeng, 1994)
The Cynefin Framework
Working with others, David Snowdon developed what on first glance appears to be another two-by-two matrix, which he called the Cynefin Framework, signifying the multifaceted nature of our environment and the influence of our experience on our understanding. The purpose of the framework was to “allow executives to see things from new viewpoints, assimilate complex concepts, and address real-world problems and opportunities” (Snowden & Boone, 2007) and it is proposed as a leader's framework for decision making.
The framework presents five contexts defined by the nature of cause and effect, which are readily applicable to the understanding of complexity in projects at a point in time (Exhibit 3). Disorder is the fifth context, shown centrally in the framework, and applicable when the nature of cause and effect is unclear. The others are:
Simple contexts, which might be equated to Type 1 projects in the Goals and Methods Matrix where goals and the methods for achieving them are well known. They are characterized by stability and clear cause and effect relationships. Snowdon and Boone (2007) describe this as the domain of best practice, and in project management terms this is the primary domain for application of mainstream project management standards. Here it is possible to categorize the project and respond accordingly but dangers arise from complacency due to initial misinterpretation of the context or because contexts can change, moving rapidly from order to chaos.
Complicated contexts, as previously described, involve multiple interconnected and interdependent parts requiring careful analysis, because although there is a clear relationship between cause and effect this may not be initially visible to everyone involved in the project. High levels of complication require teams of experts, but this represents a danger, as these experts are trained to think and see the world in particular ways and may overlook or dismiss other interpretations, opportunities, and stakeholder perspectives.
Complex contexts have the characteristics of complex adaptive systems described earlier in this paper. Snowdon describes this as the domain of emergence. Relationships between cause and effect can only be perceived in retrospect. In this realm, instead of imposing a course of action, leaders should take a more flexible and facilitative approach. Linking ideas from the Goals and Methods Matrix, they should first probe the fog to understand the context, and then respond, perhaps incrementally, or experimentally, with smaller low-risk actions or sub-projects, rather like walking carefully forward in the fog. Simple or complicated contexts may become complex due to internal or external environmental changes or, as discussed earlier, as a result of the interaction of people with different worldviews and agendas.
Chaotic contexts defy the search for right answers or cause and effect relationships because they are constantly shifting. This is an extreme case of complex adaptive systems where the leader must first act to restore some semblance of order then work to transform the situation from chaos to complexity (Snowdon and Boone, 2007). There is no relationship between cause and effect at systems level and the approach is to Act – Sense – Respond in order to discover novel practice. Snowdon describes a cliff between simple and chaotic contexts, because incorrect categorization of simplicity or complacency and failure to recognize a change in context can easily tip a project from order into chaos.
Exhibit 3 – The Cynefin Framework (Source: http://www.en.wikipedia.org/wiki/File:Cynefin_framework_Feb_2011.jpeg)
Images of Projects
Mark Winter and Tony Szczepanek (2009) propose thinking of different images of projects as a way of recognizing multiple perspectives and helping leaders enrich their understanding of their projects. One of the primary sources of inspiration for Images of Projects is Gareth Morgan's classic Images of Organizations (1997), which sees organizations as complex, multifaceted, and paradoxical, and therefore presenting challenges for managers. Winter and Szczepanek invite leaders to enhance the understanding of their projects by thinking of them from seven different perspectives:
- Projects as social processes
- Projects as political processes
- Projects as intervention processes
- Projects as value creation processes
- Projects as developmental processes
- Projects as temporary organizations
- Projects as change processes
Projects are most likely to present as specifically addressing one of these images, but most projects, and all complex projects, will exhibit each of these dimensions. Bringing stakeholders together to look at projects through each of these lenses will increase shared understanding and may assist in identifying potential risks, re-formulating the project, and changing our approaches to enhance opportunities for success.
Like De Bono's Six Thinking Hats (1999), which was another source of inspiration for Images of Projects, this approach can be used to understand the nature of complexity in the project, to develop a set of solutions, and choose a way forward.
Leadership for Complexity
Leadership has traditionally focused on the individual leader, and from this perspective the style of leadership generally considered most effective in the context of complex projects is transformative leadership. Muller and Turner (2007), researching the impact of leadership style on project success, found that medium complexity projects required emotional resilience and communication skills, and that sensitivity was important on high complexity projects.
Remington (2011), based on interviews with “good” leaders of complex projects, identified a number of common themes in terms of what these leaders did when faced with the challenges of complexity in their projects and programs. This paper has focused on providing selected frameworks that can be used to understand the nature of complexity in projects, to identify factors that contribute to complexity, and to categorize dimensions and levels of complexity. The importance of comprehending complexity is highlighted in the findings from Remington's research. Effective leaders understand the difference between complicated and complex projects and the importance of early recognition of complexity by all stakeholders.
The characteristics of complex adaptive systems exhibited by projects, such as emergence, unpredictability, non-linearity, and uncertainty, are driven by the interactions between people in projects, highlighting the key role of communication. Remington found that effective leaders of complex projects were good communicators, comfortable with uncertainty, and able to adjust their communication style to suit the circumstances. These leaders did not treat their role as a solo act. They cultivated and energized teams at all levels within and around the project.
The law of requisite variety, formulated by Ashby (1965) in the context of cybernetics, proposes that where there is greater variety there is greater complexity, and that as only complexity can absorb complexity, in order to control a system, the controlling or regulating system must have more variety than the system being controlled. Program management allows for more variety and flexibility in managerial practice than project management. Effective leaders of complex projects adopt a program management approach to their projects, which fits well with the idea proposed earlier of probing the fog and progressively realizing complex projects or programs through a series of better defined and therefore less complex projects. This way, they are able to tailor their approaches to the specific requirements of each particular activity and reduce the timeframes, which in themselves can be a source of complexity as they increase duration of exposure to internal and external interactions. They can also look beyond the individual project or projects to the delivery of benefits.
Complexity can be seen as a driver of innovation. Using different frameworks to understand the complexity in projects encourages creativity and innovation in response to the challenges of complexity, and this was a theme supported by Remington's study.
De Bono's blue hat (1999) encourages us to reflect, and to think about thinking, and Winter and Szczepanek's Images of Projects provide us with a practical, project-focused way of doing this. Remington's leaders of complex projects were reflective practitioners.
In the context of complex adaptive systems all agents, whether they be individuals, teams, or organizations, bring their own histories and cultures. Social interaction is the major source of complexity in projects, and social complexity involves both political and cultural aspects. Winter and Szczepanek's images encourage us to look beyond the technical to understand projects from organizational, political, and cultural perspectives. Understanding of cultural issues and exercising political skill were consistent themes in Remington's interviews with effective leaders of complex projects.
Finally, when crises occur and projects fall into chaos, as described by Snowdon in his Cynefin Framework, effective leaders do as Snowdon suggests and switch to a top-down, directive leadership style to pilot their projects through the crisis (Remington, 2011).
Perhaps the most important and also colorful aspects of Remington's findings was that effective leaders of projects are “humble iconoclasts.” These leaders are reflective and have high levels of self-awareness, so that they have the humility to listen, to observe, to learn, and to admit when they are wrong. This self-awareness, combined with emotional resilience and comfort with ambiguity, are consistent themes in discussion of leadership of complex projects. It also gives these leaders the courage to break the rules when necessary and the determination to achieve results against the odds.
Emergent Leadership and Complexity Leadership Theory
So far this discussion has focused on the individual leader. There are other perspectives on leadership in the context of complexity that focus more on the leadership that emerges from the interactions between people. This may be described as emergent or shared leadership where multiple leaders emerge from time to time to exercise influence over the direction of the project. Complexity leadership theory (Lichtenstein, Uhl-Bien, Marion, Seers, & Orton, 2006) takes a slightly different perspective, in that it interprets leadership not as a quality that is in a leader or something that is done by a leader, whether in an hierarchical or shared leadership sense.
Rather, complexity leadership theory recognizes that social processes are too complex and messy for leadership to be attributed to a single individual, or even a series of individuals. Relationships are not hierarchical but, instead, defined by interactions. Leadership then becomes a phenomenon of the complex adaptive system, transcending the individual, and can be seen as an emergent event rather than the quality of person. Leadership effectively occurs in the spaces or interactions between people in the project. In this context, “leaders” in the formal sense can become enablers or orchestrators of the conditions in which leadership emerges. Adopting a flexible approach, providing support, fostering trust and encouraging people in projects to appreciate the systemicity and complexity of their endeavours is one way in which a “formal” leader can release and benefit from the shared and distributed leadership that emerges from interactions.
Summary and Conclusion
One of the challenges for everyone involved in complex projects is in knowing what to do when you don't know what to do. Under these circumstances we have to learn our way through the project. Effective leaders of complex projects will address this challenge by working with stakeholders to appreciate the nature of the complexity in the project. They will be sufficiently self-aware to understand their own strengths and weaknesses, to listen, to observe, to learn and to admit that they don't have all or perhaps any of the answers. They must ensure that everyone remains vigilant to unexpected and kaleidescopic changes occurring within the project as a result of interactions between agents (people, teams, organizations) and with the environment.
Many tools are available to help in the categorization and comprehension of complexity, and a small number of them have been briefly described in this paper. Their major value is in providing a basis for discussion and shared understanding of the nature of complex projects and the challenges that each one presents.
Ashby, W. R. (1965). Introduction to cybernetics. London: Chapman & Hall.
Baccarini, D. (1996). The concept of project complexity—A review. International Journal of Project Management, 14(4), 201–204.
Cilliers, P. (2000). What can we learn from a theory of complexity? Emergence, 2(1), 23–33.
Crawford, L. H., Hobbs, J. B., & Turner, J. R. (2005). Project categorization systems: Aligning capability with strategy for better results. Newtown Square, PA: Project Management Institute.
Crawford, L. H., & Pollack, J. B. (2004). Hard and soft projects: A framework for analysis. International Journal of Project Management, 22(8), 645–653.
Crawford, L. H., Sankaran, S., & Butler, Y. (2005). Project management in complex times. Proceedings of the PMOZ Conference, Brisbane.
De Bono, E. (1999). Six thinking hats. Boston: Bay Back Books, Little, Brown and Company.
Lichtenstein, B. B., Uhl-Bien, M., Marion, R., Seers, A., & Orton, D. (2006). Complex leadership theory: An interactive perspective on leading in complex adaptive systems. Retrieved from http://digitalcommons.unl.edu/managementfacpub/8/
Morgan, G. (1997). Images of organization. Thousand Oaks, CA: Sage Publications.
Müller, R., & Turner, J. R. (2007). Matching the project manager's leadership style to project type. International Journal of Project Management, 25(1), 21–32. Retrieved from http://www.sciencedirect.com/science/article/B6V9V-4MNRD0K-3/2/1c45beedf5ba8efdbca5607aaa30cdd2
Obeng, E. (1994). All change! The project leader's secret handbook. London: Pitman Publishing.
Remington, K. (2011). Leading complex projects. Farnham, Surrey: Gower Publishing Limited.
Snowden, D. J., & Boone, M. E. (2007). A leader's framework for decision making. Harvard Business Review, 85(11), 68–76. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=bth&AN=27036324&site=ehost-live
Snowdon, D. (2002). Complex acts of knowing: Paradox and descriptive self-awareness. Journal of Knowledge Management, 6(2), 100–111.
Turner, J. R., & Cochrane, R. A. (1993). Goals-and-methods matrix: Coping with projects with ill defined goals and/or methods of achieving them. International Journal of Project Management, 11(2), 93–112.
Vidal, L.-A., & Marle, F. (2008). Understanding project complexity: Implications on project management. Kybernetes, 37(8), 1094–1110.
Williams, T. M. (2002). Modelling complex projects. Chichester, New York: Wiley.
Winter, M. C., & Smith, C. (2005). Making sense so far: Emerging directions and future research: Interim Report.
Winter, M. C., & Szczepanek, T. (2009). Images of projects. Farnham, Surrey: Gower Publishing Limited.
© 2013 Lynn Crawford
Originally published as part of 2013 PMI Global Congress Proceedings – Istanbul, Turkey