Main Insight
Most of the problems companies face in developing products and services started out as small errors that were undetected and become large ones. Tight coupling of work in a company often has these small errors cascade throughout the organization over time. The cause and effect underlying all of this is often obscured by complexity. However, instead of worrying about how complex a system is, these challenges can be avoided by creating visibility via feedback and decoupling scheduling problems via using pull.
A focus on creating feedback and reducing coupling is more useful than worrying about the degree of complexity in the system.
Article
There is a lot of conversation about complexity in the Agile community. Discussions about states – complex, complicated, simple and chaos abound. But focusing on state (complexity in particular) may be a red herring – a distraction from what we really need to pay attention to – chaotic events.
Chaotic events are epitomized by the “butterfly effect.” A small event causing a very large one. The proverbial “straw that broke the camel’s back” is another example. These events are also called “non-linear events” because if the impact is plotted over time we get a small aberration leading to a big jump in impact. Note that chaotic events may lead to chaos, but one is an event the other is a state.
One of the reasons waterfall does not work well is that it obscures the effects of many chaotic, non-linear events, until late in the project. These include small misunderstandings, short delays in capacity availability, and slight misreadings of the market. The net effect is a massive impact on schedule and quality. This can be seen in scheduling as well. While we often lament a project being delayed by months, on closer examination it can be seen that it was really an accumulation of short delays throughout its development creating a cascading (coupled) effect.
In Meltdown, Clearfield and Tilcsik explain how what started as a plumbing problem coupled with a stuck value and lack of clarity due to an ambiguous indicator light led to the Three Mile Island disaster. On reflection of this and other disasters, it can be seen that a small event tightly coupled to other events that have a larger impact, may cause a very large problem. This only occurs, of course, when left unchecked because these interactions are not visible.
When analyzing what causes failures in product development we see the combination of these same forces:
- a slight misunderstanding of a requirement
- results in it not being as effective as it should be
- no one notices this until this misunderstanding has propagated into the field.
Much of Agile is about how to be, but it also incorporates doing things in smaller steps. An implicit purpose of Agile is quick feedback which can prevent chaotic events from happening while simultaneously improving quality.
We need a shift from looking at states to looking at events. Complexity may dissuade us by doing so because we may believe there is no way to see events as they unfold. But cause and effect isn’t even needed here, visibility via feedback is.
In our workflow, potential chaotic events are all around us – misunderstandings and coding/design errors being two very common ones. We must work to reduce the impact of these with quick feedback cycles. The question is ask is “what needs to be present in order for a chaotic event to occur?” The answer is coupling and lack of visibility.
Decoupling of cascading scheduling problems can often be ameliorated by the use of a pull (Kanban) mechanism. Lack of visibility can be eliminated through a combination of test-first, feedback and automated testing.
All of this applies to any work we do. This includes attempts at improving our workflows. Again, we must also attend to chaotic events. Making a change that seems obvious may, in fact, cause problems that are not seen until the damage has been done. Again, visibility and decoupling can help here.
Regardless of your approach, change via emergence is important. Make a change, see what happens and get an improvement or learn something to prepare for your next move.
We can think of this combination of potential chaotic event, coupling and lack of visibility as gunpowder. Gun powder consists of a fuel (charcoal), an oxidizer (saltpeter or niter), and a stabilizer (sulfur) to allow for a constant reaction. If you remove the oxidizer (lack of feedback in our situation) you have a harmless mass of charcoal and sulfur.
We don’t need to eliminate the complexity, just deal with its ramifications. In other words, you don’t need to manage complexity, you manage what makes complexity risky – lack of visibility..
Ironically, complexity sounds so daunting at times that people avoid trying to go beyond it. These quotes from Dr. Goldratt in The Choice* provide some insight.
The first and most profound obstacle is that people believe that reality is complex, and therefore they are looking for sophisticated explanations for complicated solutions. Do you understand how devastating this is?
The more complicated the situation seems to be, the simpler the solution must be.
* Note that Dr. Goldratt is better known for The Goal. But The Choice (‘2008) is just as iconic, highly recommended and guaranteed to change your ideas on complexity. Don’t be misled by some people’s misinterpretation of Goldratt’s work being applicable only to manufacturing and complicated systems. Many people are taking his work into multiple fields in the same way they and others are taking Lean and the Toyota Production System outside of manufacturing.
In addition, ignore those who dismiss Dr. Goldratt’s notions of inherent simplicity believing they date back to the 80s, when, in fact, The Choice was first published in 2008.
For more on applying Dr. Goldratt’s thinking to knowledge work see Dealing with Complexity by Creating a Bias For Simplicity