Untying the Gordian knot of complex projects -- a structured approach to complexity
For many years, the author has witnessed projects in their organisations, and in those of their clients, fail for one spectacular reason or another. In some cases, the project manager was inexperienced, in others, the stakeholders simply could not agree on the project’s objectives, and, in many too numerous to recount, scope creep caused the project to take on such an amorphous shape it was simply too difficult to “get one arms around,” or, stated another way, control.
Yet, in most, if not all, of these spectacular failures, one could find as a root cause the inability of the project manager, team members, client, sponsor, or associated stakeholders, to discern the level of complexity with which they were faced. Much like the mountain climber who, in looking up to fasten the next karabiner, sees an avalanche fast approaching, project managers often recognize the complexity of their undertaking far too late to effectively address it. In short, they are buried under it and spend an inordinate amount of time fighting the long, uphill battle to gain control.
So, how does one recognize complexity early enough in the process to do something about it? And, what exactly is that “something” that needs to be done? This paper addresses those questions by looking to the science of complexity.
The Science of Complexity and Project Management
A revolution is taking place in science today. That revolution is the emerging science of complexity that crosses all the borders of tradition. It applies to fields as diverse as topology, economics, physics, and biology. This science asks us to examine some long-held beliefs about the systems we manage, our understanding about how the parts of those systems relate to each other, how information flows through the system, and the need for cooperation among the parts.
Project management is a unique competency that requires specialized skills in integration, scope, time, cost, quality, human resources, communication, risk, and procurement management. The challenge for project managers is to manage many or all of these areas in relation to each other. Historically, project managers have attempted to manage challenging projects by using control-oriented techniques, executed in a rigid top-down hierarchical structure that emphasized compliance, conformity, and adherence. In complex projects the problems faced by traditional project managers are magnified because the variables are too many to identify. If we cannot identify them, then how can we schedule them? How can we cost them? How do we know what the scope is? These are the nonlinear factors that occur on every project and cause our projects to suffer from scope creep. Yet, our traditional solution has been to apply a linear approach to estimating project schedule and budget. We disregard the other factors because they are unrealistic or too difficult to quantify, but they are factors that occur on every project and on all the tasks.
Our approach begins with an understanding of complexity as represented in ESI’s Complex Project Management Model, a perspective on the topic we developed during the past several years, along with several of our colleagues, to help project managers prepare themselves for dealing with the really difficult projects. Additionally, we are taking the additional steps of developing the ESI “complexity indicator,” a straightforward tool used to aid the project manager and his or her team in determining the level and nature of project complexity. Once fully developed, the tool will also help project managers unravel the “mysteries” of current projects to get them back on track.
This structured approach to complexity, therefore, is designed to focus the project manager’s attention on those activities that could derail the project at the outset. It is not a panacea, nor is it a guarantee. It is, however, a start and a needed one because large-scale projects are among the most important and consistently mismanaged endeavors in modern society. But first, a bit of history.
The Past as Prelude
Where did our ideas about managing originate? Are the ideas still valid in the modern enterprise?
As managers, we are confronted with increasingly complex projects. Additional factors of reduced resources and an emphasis on speed-to-market add more pressure. Why are these efforts failing or are less than optimal? The answer is that every organisation and project manager are not examining projects and practices periodically with a view toward renewal or self-improvement.
Managerial positions were created to manage the new factories created as a result of the Industrial Revolution. But the lack of management expertise caused factory owners to seek advice and guidance from outside experts and philosophers. The work of three philosophers played particularly important roles in shaping management thinking that is prevalent to this day.
The first of these was René Descartes, a mathematician and philosopher who is famous for the concept of reductionism. Reductionism is the idea that to better understand something you must break it down into component parts. If you understand the parts, then you understand the whole. The second was Sir Isaac Newton whose ideas about the mechanistic universe extended the theories of Descartes and gave us the concepts of serial linkage and machine-like organizational structures. Lastly, Adam Smith wrote Wealth of Nations, advocating a “division of labor” as a method of control and efficiency. The ideas of these three thinkers helped established the management techniques used during and after the Industrial Revolution.
In its time, machine-age management practices served their purpose well. However, these practices consider the worker to be essentially utilitarian. But workers have become better educated, more socially aware, and exposed to more information, making this utilitarian view nonviable. In the modern project environment, workers often generate solutions to complex problems. In large, complex projects the need for worker input and commitment becomes even more important. Many theorists suggest that the internal workers’ commitment is critical for success on a typical project and mandatory for success on a large, complex project. Though technical know-how is an admired trait of many project managers, it is difficult for project managers to have the technical subject matter expertise for all aspects of complex projects. Some project teams also may not possess the technical expertise necessary to create full and detailed project plans.
The Future in Present Form
The nature of many of today’s projects and much of our work in the information age is increasingly dynamic and complex. Projects and business processes have many—
- Interested and participating parties;
- Technical variables;
- Geographic and cultural diversity issues;
- Economic variables;
- Time variables;
- Legal variables.
In an effort to deal with dynamics and complexity, managers have resorted to increased control, rules, competition, and other Newtonian concepts that have proven to be ineffective or of limited value. To manage complexity effectively, managers must understand complex systems and how they affect projects, products, and processes.
ESI’s Complex Project Management Model
ESI’s model has four components:
- Knowledge of complexity: the science that helps people understand the interactions that occur on a project.
- Communication: the methods of communication, critical information that must be transferred, and the essentials of feedback.
- Leadership: the role of the project manager in guiding and managing the project and related interactions.
- Project management: the structuring of the project organization, and the procedures used to manage complex projects.
ESI’s model represents a systemic approach, as show in Exhibit 1. The effective management of complex projects requires the engagement of all the component parts of the diagram. Each part reinforces and supports the others. The assumptions in designing the model are:
- It is a departure from typical models in that it sets a mental framework, but is not process or flow oriented.
- Each complex project possesses its own unique elements and remedies. Just as complex projects are highly interrelated, so is the model for their effective management. One cannot effectively manage complex projects without all the parts being represented in the model.
- Leadership is the core—and forms the heart—of the model.
Exhibit 1: ESI’s Complex Project Management Model
Let’s now explore more fully the four parts of the model.
Knowledge of Complexity
The first bubble in the model is the knowledge of complexity that must be understood first as it helps shape activities and thinking for the other components.
A key area in understanding complexity is the penchant for people to self-organise. Self-organisation is the evolution of a system into an organised form in the absence of external constraints (just think of teenagers at home on Saturday night when the parents are gone!).
Self-organisation is the concept that new relationships, methods, and activities emerge in unpredictable ways on projects and is the most critical of the fundamental principles in complex projects. The phenomenon of self-organisation is especially important because it is a source of frustration and conflict for project managers and team members who do not understand it; and as projects become larger, and the number of project participants increases, the effect of self-organisation increases exponentially. Disperse those team members around the country, or around the world, the effect of self-organisation is exacerbated!
In addition to the concept of self-organisation, there are four other complexity issues that project managers and team members must understand. They include:
- Nonlinear dynamics: A process or series that does not flow in a straight, predictable manner. A nonlinear process is one that is interrupted repeatedly by events, reorganisation, new information, and the like.
- Open/closed systems: The open, self-organizing system uses energy, material, and feedback (information) from its internal and external environments to organise and reorganize itself. A closed system is one that ineffectively interacts with other systems or parts of the system, does not process information efficiently, and becomes less and less effective as a participant in the overall environment.Project managers who create open systems will be more successful in the complex project management environment than those who create closed systems.
- Emergence: The concept that parts of the project interacting with each other can form a bond, relationship, or capability that the parts of the system did not have on their own.
- Human behavior: Understanding human behavior is key to successful project management. Every individual brings behavioral characteristics to the project team environment. The project manager—more than any other team member—must understand how to interpret and channel the team’s different personalities.
A common practice in the machine age was to communicate on a need-to-know basis. In the modern era, however, project members and stakeholders must have a solid understanding of project issues. Study after study of why projects fail show that they do so not because of technical problems but due, in part, to the following:
- Poor communication;
- Ineffective start-up practices;
- Ineffective project organisation;
- Ineffective implementation of procedures and processes.
Restrictive- or control-oriented approaches are ineffective in the complex environment. The chart below contrasts the machine-age management techniques to those recommended for effective complexity management.
Exhibit 2: Effective Complexity Management Chart
The technical details of the project will be managed effectively at the appropriate level within the work breakdown structure. The project manager and the project team’s task is to—
- Ensure uniform and consistent performance;
- Develop a communication system that eliminates boundaries and enables information sharing and collaboration.
Communication in the complex project environment cannot be haphazardly conducted. It must be the result of a well-conceived plan. The plan should address the following concerns:
- With whom must there be communication?
- What information needs to be transferred?
- Are there unique needs (language, cultural, geographic, interoperability, and so on) that should be incorporated into the plan?
- How often must planned communication occur?
- What are the guidelines for communication?
- What are the specific contact methods?
- Can the communication system adjust to self-organization at the edge of chaos?
Communication is the first half of the information process—managing the information is the second half.
Information can be communicated either formally or informally, but both need to be captured and managed. A formal system for gathering, integrating, and distributing critical project information is called the project management information system (PMIS). A project office or a project assistant normally manages the PMIS database. Corporate standards or standard operating procedures usually determine the type of information that is collected and reported. It is a good project management practice to recognize that large, complex projects will generate volumes of information and data, and that a coherent approach to information and data management is critical to the project’s success. The PMIS will typically define what information is required to be reported to the project’s critical stakeholders.
Complex projects place special demands on managers. The project manager must have knowledge of:
- Technology of the project;
- Project management;
- The organisation for which they work;
- The customer organisation;
- The marketplace;
- Project team members.
But more important than one’s ability to manage a complex project, is one’s ability to “lead” a group of people in the project management of the complex project. Much has been written on the difference between management and leadership; and, John Kotter, a popular lecturer and writer on organisational dynamics from Harvard University, has weighed into the debate with his belief that management and leadership have distinctly different goals and are very different behaviors. The behaviors required for complex projects are more along the lines of “leadership” rather than “management.”
Too often mangers of complex projects desire to run their projects according to the Newtonian mental model and military command and control structure. In the former, the structure was based on the pursuit of reliance, stability, and predictability, while the latter is characterized by an organisation that is tightly linked and fine-tuned and everyone salutes and does what they are ordered to do. Complex projects, and we would dare say, projects in general today, can be described as living organisms that are evolving and ever changing, and are indeed more fluid than many of us would like them to be. Therefore, unpredictability, instability, uncertainty about the future, vast amounts of information, and the changing workforce mandates another set of behaviors for success.
The role of the leader, as defined by Kotter, is clearly more consistent with the behaviors necessary for complex projects. Words such as supportive, collaboration, principles, guide, inclusive, open, decentralized, and sharing describe a pattern of behavior different from those often used to describe the other models discussed above. It would be helpful to contrast this understanding to the behaviors described by Kotter as shown in the chart below. (Kotter, 1996)
Exhibit 3: Roles of Leaders and Managers
In complex projects we need both sets of skills in our project managers. The concepts of leadership do not replace the Newtonian ones; in fact, they complement them. Organisations such as Citigroup, the U.S. Army, Motorola, IBM, American Express, and many others have recognised that these new concepts are integral to success.
It is in the fourth component of the model that the project manager effectively employs his or her understanding of complexity, strong communication skills, and leadership ability to get the job done. As regards project management, project managers of complex projects must be able to:
- Define and implement a project structure that enables the accomplishment of the new vision and project charter or contract.
- Help the project team members achieve the triple constraints while encouraging interaction.
- Establish a well-define plan that functions as an “area of order.”
- Select and implement project procedures that promote organsation while, at the same time, enabling appropriate actions on the edges of disorganization (the edge of chaos).
- Establish a communication framework, structure, and process, to ensure everyone has the information they need, when they need it, to accomplish project objectives.
While we can look at this list and remark that any project manager must have these skills and be correct, the issue in complex projects is the extent to which the project manager exercises his or her leadership ability across a broad spectrum of participants. Ultimately, the major difference between complex projects and other projects lies in the leadership component; as projects become larger or more complex, or both, the degree if success is directly proportional to the project manager’s leadership ability and the intangible characteristics that often define leadership.
The ESI Complexity Indicator
Based on the model described above ESI has developed, and continues to refine, the Complexity Indicator, a questionnaire that aids project managers in determining the extent to which their project is complex. The questionnaire is a single-question, multiple-response format, paper-based (at present) document. This tool provides a finite number of questions oriented around the project areas that are affected by complexity. The scoring will indicate the project’s complexity rating on a scale of 0 to 10.
The score will determine the entry into an “action matrix” that recommends certain approaches, techniques, and tools to lead the project and manage the complexity that is under development. There are, for example, specific actions that can be taken to manage the project’s complexity thereby reducing the project’s complexity score on subsequent indicator applications. The indicator asks questions based on the project management process to include initiating, planning, executing, controlling and closing a project.
The authors wish to thank Paul Lombard and Chip Stillwell for contributing their insights to managing complex projects based on complexity science as well as Jim Taylor and Jim Foreman of ESI for their additional insights into this exciting field.
Kotter, J.(1996). Leading change. Boston: Harvard University Press.
Sterman, J. D. (2000). Business dynamics: Systems thinking and modeling for a complex world. New York, McGraw-Hill Higher Education.
©2005, ESI International
Originally published as a part of 2005 PMI Global Congress Proceedings–Singapore