Disciplined Agile

Dealing with Complexity by Creating a Bias For Simplicity

Table of Contents

It is difficult to make predictions, especially about the future. – Mark Twain
For every complex human problem, there is a solution that is neat, simple and wrong. – H. L. Mencken

Executive Overview

We live in a VUCA world (volatility, uncertainty, complexity, ambiguity). It is important to understand that organizations are not simple, nor even just a collection of components that can be understood and managed. Even when the laws of Flow, Lean, Theory of Constraints, organizational development and more are understood, the number of interactions in a company are over-whelming. The key is not to take complexity head on but to understand that systems are really systems within systems. Some of these systems are fairly predictable but are wrapped in an informal structure that makes perceiving what is happening difficult if not impossible.

The first step to improve our methods requires attending to the natural laws present in our work. An example of a “natural law” is putting people on too many projects increases multi-tasking, injects wasted work into the system and delays the delivery of value. By avoiding violating these laws, we can move forward. Often in emergent steps as planning very far ahead is not usually possible. Sometimes these steps achieve improvement. When they don’t, they illuminate previously unseen or understood relationships in the system. This learning opens up new options for us.

This page discusses how to look at our organization’s development method to see what we need to improve. Armed with these insights, we can move forward on the other challenge – implementing these improvements. Having clarity on what will work and why enables us to be more able to overcome the natural resistance to change that is ever-present.
This page lays out:

  • Inherent simplicity – the belief that inherent in complex systems is a simple model to attend to that will enable effective action
  • Factors for simplicity – what to look when you’re company is doing product or service development
  • Creating and adopting a plan – how to use inherent simplicity to improve your organization’s effectiveness.

 

Inherent Simplicity

It’s well established that fully discerning cause and effect in complex systems is not possible. Improving systems is therefore not fully predictable and relies on emergent methods. But there are also natural laws embedded in systems that, when violated, cause problems. Attending to them is one key to making effective change.

One trick to discovering these is not to look at how to improve things, but to look at how to make them worse. For example, noticing that doubling workload for busy people will make things worse identifies the relationship between workload and productivity. Considering delaying feedback from customers or isolating people from each other again reveals causes and effects.
Stopping the violation of these natural laws is a critical step forward. But we must differentiate between what we want to do, and the how we will accomplish it. While we may have clarity in the first, the path forward will be confronted by volatility, uncertainty, complexity, and ambiguity (VUCA).

The Theory of Constraints puts forward the concept of inherent simplicity – the presumption that inherent in complex systems are rules that, when understood, enormously simplify how to look at the system. These rules already exist. We must find them and take advantage of them. Doing so enables us to increase performance and reduce or eliminate the challenges we are facing.

When we apply inherent simplicity through the eyes of Flow and Lean we see our workplace in a different light. Instead of looking for the complexity in the problem domain, let’s look at it from the perspective of Flow and Lean. We are not trying to look at our world through the principles of Flow and Lean. That would still be too much to do. What we want are a few dimensions that, when we explore them, expose most, if not all, of the significant issues commonly present.

 

Factors for Simplicity

If we dive deep enough we’ll find that there are very few elements at the base – the root causes – which through cause-and-effect connections are governing the whole system. The Choice, Dr. Eli Goldratt.

Inherent simplicity tells us we need to look at our problem in a particular way – one that is inherently defined in the problem itself. These “factors for simplicity” can be considered the dimensions of our problem, that is, factors we can look at that enable us to understand our problem. For example, when building a bridge, one must consider the strength of materials (elasticity and compression), wind and a few other factors. These factors for simplicity will frame how to look at our practices. We can both see what happens when we don’t attend to them and we can see what we can do when we attend to them.

For each of these we also need an understanding of the natural laws that affect them (e.g., gravity). We can learn how to guide the actions we can take by understanding useful principles that can achieve our goals (e.g., Lean and Flow thinking). Part of this will be being able to measure how well we are doing (e.g., the value stream impedance scorecard).
There are several lessons from Lean and Flow that are particularly useful in deciding what to do. The following list is what we’ve found useful for developing products and services:

  1. focus on customer value
  2. use small batches
  3. attend to the value stream
  4. eliminate waste by removing delays
  5. make all work and workflow visible
  6. manage work in process
  7. build quality in

From these we derive our factors for simplicity. The relationship to the above list should be clear:

  1. The value density of the items being worked on
  2. Batch size of work
  3. Effectiveness/efficiency of the value streams
  4. Visibility of work and workflow 
  5. How workload relates to capacity
  6. A minimal number of interruptions from outside the value stream
  7. The at which we get feedback
  8. Quality of the product
  9. The value creation structure of the organization

It is important to understand the true nature of the system we are in and how we can see to change its behavior. We risk waste or ineffectiveness if we don’t understand what we are actually dealing with. The following table lists several dimensions of inherent simplicity and what actions can be guided by them.

While we can start with these examples, it is important to not fall into the trap of using these as mantras to follow and not continuing the exploration of the inherent simplicity in your system.

While giving acknowledgement to Theory of Constrains for the term “inherent simplicity”, we prefer to call what we’re trying to do a bias for simplicity.  Instead of looking at our problem as intrinsically complex, we want to look for a way to view the system in a way that we shed light on its behavior by looking at the problem in a simpler manner. This also has us not interpret Dr. Goldratt’s work in a manner he hadn’t intended.

 

Other Important Factors

There are, of course, other factors than just these factors for simplicity that affects our work. The culture of the organization is a major one. How management works with the talent is another.
 

A Warning

Things should be as simple as possible, but no simpler.  Albert Einstein

We are not looking for simple solutions. We are looking for a way to look at our problem so that the problem becomes simple and our solutions become effective.

 

Using Factors for Simplicity, Natural Laws and Actions

  1. Factors for simplicity provide us a way to look at a complex system in a simple manner
  2. Natural laws are present in the system and affect each component of the system and the relationships between the components
  3. We can learn principles of Lean, Flow and other methods to understand how to work with these natural laws
  4. We can organize solutions as sets of patterns so we can see how actions in one part of the organization relate to others
  5. We can create a collection of specific solutions that exist in these patterns so that we don’t need to reinvent the wheel but can still create solutions that are specific to our needs

The above list also provides insights into how FLEX and Disciplined Agile relate to each other – FLEX providing us with a powerful way of looking at the first four items in the list and with Disciplined Agile providing us with the wealth of options needed.

 

Why A Bias for Simplicity Is Important

The first and most profound obstacle is that people believe that reality is complex, and therefore they are looking for sophisticated explanations for complicated solutions. Do you understand how devastating this is?” The Choice, Dr. Eli Goldratt.

The ability to predict systems as a whole has been demonstrated many times over. However, one does need to understand the difference between micro-predictability (e.g., predicting the color the ball in a roulette wheel will land on) and macro-predictability (e.g., more money is staying at the table).  But this predictability comes from discovering the predictable factors embedded in the complex system. In knowledge work these are typically defined by laws of Flow, Lean and Theory of Constraints.

In “The Choice”, Dr. Goldratt, creator of the Theory of Constraints, presents the presumption that inherent in complex systems are rules that, when understood, enormously simplify how to look at the system. These rules already exist. We must find them and take advantage of them. Dr. Goldratt calls this  Inherent Simplicity.  He suggests we must recognize the need to come from is how to look at complex systems.

This is not just a better approach, it avoids a significant danger of not doing it. Thes reverence for complexity is deadly. Again, from Dr. Goldratt – “The first and most profound obstacle is that people believe that reality is complex, and therefore they are looking for sophisticated explanations for complicated solutions. Do you understand how devastating this is?” The “devastation” results from believing a system is complex and then looking for sophisticated explanations for complicated solutions. It’s the search for sophisticated explanations that’s devastating – because simpler explanations  are available but not being investigated.

This isn’t to say that the world isn’t complex. From The Choice – “He doesn’t claim that reality is not overwhelmingly complex; he acknowledges it in full. But what he says is that there is a way to realize that from another more important aspect, it is exceedingly simple”
We already have a great deal of experience with a bias for simplicity. It is often called “intuition.” When we make an endeavor to discover this consciously, we become more adept at it and can greatly increase our understanding of what we need to do.

 

We need feedback, no matter how well we think we understand things

The potential of mis-understandings and other potentially chaotic (small error big damage) we must always get feedback on our actions. Therefore, however, confident we are, we must always be suspicious of our actions and use feedback to ensure we’re getting the results we want. We should also take the attitude that we are adding value when we learn.

If we don’t get the result we anticipate, we should ask why? It may be that we are in a part of the system that is unpredictable, but much more likely is there is a relationship that we either weren’t paying attention to or misunderstood. We can use the results we achieved (good or bad) to improve our understanding of the system.

By taking this attitude, is that even when we don’t get the result we want, our understanding always improves.

 

Using the Simplicity Factors to Predictably Achieve Good Results

You cannot get full predictability in what results you will get from attempts at improvement. We can get better results in our improvement methods if we recognize that attempts at change have various aspects to them, each with their own degree of certainty. These are:

  • The proposed change itself
  • Whether the change is theoretically beneficial
  • The effects of it being implemented
  • The anticipated cost of doing it
  • Getting people to do it

The proposed changes can be made by looking at the factors for simplicity and seeing what needs improvement. It might be surprising, but many practices espoused by popular frameworks violate several laws of flow and lean.  This is often because they are applied universally, instead of attending to the context they are being used in. Before blindly following any practice (even if you came up with the idea) it is worth seeing whether it follows or violates principles known to be useful.

Given the uncertainty present, we should be aware of the anticipated, and potentially unanticipated effects of the changes’ implementation. The way people work is more defined by how work done in different parts of the organization relate to other parts. This is a common characteristic of complex systems – they are defined by the relationships between their components more than the components themselves.  Since many relationships are hard to see or understand, we must see what actually happens and adjust accordingly. Thus, while we can have reasonable certainty of the theoretical improvement, we must attend to the actual result.

The greatest uncertainty is often getting people to actually do it. The longer it takes and the greater the resistance to doing it, the greater the cost will be.  By using the factors for simplicity, we can demonstrate the why for the suggested improvement. This can improve alignment across the organization and create a greater focus on achieving a global improvement and not just local optimization.

In complex systems you cannot usually get fully predict what a solution to your problems will be. The intent is not so much to just jump to a good solution but to follow a path that will get you to one. The simplicity factors provide a mechanism for this:

  1. By looking at our system by attending to the simplicity factors we’ve identified – workload, efficiency, batch size, collaboration, management’s role, planning and quality – we can readily see what our greatest impediments are
  2. How we are violating the natural laws of Flow and Lean should now be apparent.
  3. We can now decide what solutions to try. It is likely that several impediments will have been discovered so we don’t want to try all of them. Doing so will not only cause cognitive overload but will make it if you’re really working on the most important things. Keep in mind that solutions further up the value stream have a greater impact on things later in the value stream than the other way.
  4. Implement what looks like a good solution and see what happens

By having clarity on what we want to do, we can great alignment in actually achieving it.

 

A Parallel with Managing the Complexity of Software

Large software systems are usually complex due to a combination of poor design/architecture and technical debt. The factors to attend to to reduce this complexity is well known (cohesion, coupling, encapsulation, testability …) along with methods to achieve these (design patterns/principles).

We would never take a “probe-sense-respond” approach with software. Instead we attend to design qualities and use automated testing to validate our changes.

This is the equivalent of using inherent simplicity.  We must, of course, validate our changes (reduction in cost-of-delay and/or cycle times are good measures). But if they don’t result in an improvement, they will result in learning. It should never be fail fast, but rather learn fast.

 

Navigating Complexity

There’s an apparent dilemma facing people attempting to improve the process of their organization. On one hand, we know doing certain things would be beneficial. On the other hand, we know nothing is really certain.

Knowledge work takes place within a complex system. Systems are defined more by the relationship between their components than by the components themselves. Consider how putting a bigger engine in a car may actually make it unusable if the frame can’t support it or the brakes can’t stop it.

Systems require an holistic view of how things relate to each other. Some relationships are clear, but others are often misunderstood or not event visible. Add the likelihood of small errors having big costs (e.g., misunderstandings) & certainty goes out the door.

It is not possible to just say “do this” & know it’ll happen, or even if the “this” is really to our benefit. Short feedback cycles of learning & doing attend to both of these uncertainties.

We should act based on known relationships and attend to the results we get to make other relationships visible & more understandable. We learn as we go via a combination of theory and practice and validation. We don’t view backtracking as failure, but as learning something about how our system work.

It is useful to think of the relationships in a system as being of the following types:

  • We can see it and we understand it (understood)
  • We can see it but we don’t understand it (not understood)
  • We can’t see it, but if we did we’d understand it (knowable)
  • We don’t see it and wouldn’t understand it even if we did (unknowable).

Types of relationships

Attending to the relationships in systems

 

Patterns and Complexity

This section provides some insights by Al Shalloway for the interested reader about his path to adopting inherent simplicity.

I have long believed in looking at patterns. Christopher Alexander’s work, in particular, The Timeless Way of Building, has had a tremendous impact on how I approach design issues.  The architecture of an approach/framework/method is just as important as the architecture of a software application. Questions such as how it is to be used, how it can accommodate change, and how it enables the user to learn more about are just a few of the questions that need to be considered when designing an approach.

Patterns taught me to look at the forces to be resolved in a system. THis is even more than the practices in the system. A key tenet of systems thinking is that relationships between components are more important than the components themselves. My ability to digest a lot of disparate information and make a cohesive theory has long enabled me to go into organizations and see both what is happening and what to do. Admittedly, I’ve had a lot of training in this – mathematics, psychology, engineering and science. Some people told me I’ve just got a knack for it. But I’ve always felt that what I was doing was bringing my instinctual observations up to conscious decisions.  This meant others could do this if they knew how to look at the problem. Inherent simplicity greatly facilitates that.

My first attempt to do this was to have a few solutions under my belt to use when appropriate. But as I noticed more patterns of somewhat local solutions I realized I could create entire system solutions from them. This led to what I called “the inflection point system” which was a collection of options for most all of the problems organizations faced in product development/maintenance that had a software component. This had a decent track record (about 50%).  Some of its failings was due to people sabotaging it. I realized that knowing what to do may be clear, but getting people to do it was another matter altogether.

I had already been doing Lean for 20 years so I knew there were what I called “natural laws” of development.  These were concepts like big batches cause problems, multi-tasking creates additional work. This was useful for insights, but it still seemed to focus on solutions.  People however, need to see a bridge between where they are and where they want to go to make it real for them. Just showing them the promised land where they don’t understand where they are doesn’t make the needed connection.

When I first came across Dr. Goldratt’s inherent simplicity I misunderstood it to be the same as the natural law view I was doing. But it isn’t. Dr. Goldratt’s intention for inherent simplicity is to understand that what appears to be a complex system, can be viewed via just a few insights. By understanding where you are it is easier to enroll others in seeing where they should go. This simpler way of viewing the system does not mean we can take a mechanistic approach. People and systems are still complex and unpredictable. Improving behavior in an emergent manner is the best we can hope for. But it does mean we can have clarity on what would make improvements or would provide clarity on what would do so.

To learn more about Inherent simplicity, watch this six minute video – Inherent Simplicity by Dr. Eliyahu M. Goldratt

 

Further reading

  • Art of Action: How Leaders Close the Gaps Between Plans, Actions and Results. Short re-cap of this great book by Stephen Bungay. This is a very pragmatic book on how to improve action.
  • The Value Stream Impedance Scorecard. The factors for simplicity can be used to determine how much resistance (impedance) to flow there is in our system.
  • The Choice. Dr. Eli Goldratt. While it seems most everyone is familiar with The Goal, The Choice is just as powerful and mind shaking. It presents an alternative to working with complex adaptive systems than feeling somehow unable to navigate the challenges present. In addition to inherent simplicity, it presents guidelines on how to do critical thinking that is worth the investment in the book alone.