There are natural laws to attend to
Many of the challenges organizations face are due to them violating the natural laws of software development. These include – lack of visibility, working on too many things, overloading staff, not having clarity on what’s most important to work on, not working on the items of greatest value, working on initiatives in too large batches, micro-management, and many more. This is somewhat akin to driving on the wrong side of the road. While you are never sure what will happen if you drive on the correct side, you can be sure it’ll be better.
There is a difference between learning how to do something better while attending to natural laws and learning how to stop ignoring natural laws.
A thought experiment to help us decide on an effective course of action
Consider how your organization is currently developing products and/or services. Now, just as a thought experiment, consider how to make things worse. For example, have people work on too many things, don’t have a process where you resolve multiple requests, have product managers create requirements for the team and just hand them off in written form, don’t get or give feedback to developers while they are working, offshore testing, have developers and testers be in separate organizations, work on large batches of work, have managers makeup schedules and then berate developers when they are not met, … You get the idea. Now, remember, don’t actually do this, just imagine what would happen.
I’m pretty sure you can see that things will get worse. Now, consider where in your organization you are doing some of these ineffective things. Could you stop doing some of them? Even partially? If it became worse when you started doing these wouldn’t it get better if you just stopped doing them? Sometimes improvements are easier to achieve just by stopping doing things incorrectly. As we go through the options in this chapter, keep this in mind. That sometimes improvements are easier to achieve just by stopping doing things incorrectly.
Of course, making these changes is no guarantee of improving your process, but generally, improvement will occur if you do them. As always, you must run any improvement as a hypothesis of it being a better way and validate that it is.
The interested reader may want to jump ahead to these chapters that go into these concepts in-depth:
The root cause of our challenges
Almost all challenges in our organization result in some violation of the natural laws of software and/or product development. It may be that there are valid reasons for this, such as not having sufficient people with the appropriate skills. But our intent should be to avoid violating these laws to the best of our abilities. This doesn’t happen for several reasons:
- they are being violated because people are unaware of them
- they are being violated because, although people are aware of the laws, they are not aware that they are violating them
- the are being violated for selfish, or local-optimization reasons
- people don’t understand how not to violate them
Actions that help solve an organization’s challenges
I am not inferring that our challenges can be solved by mechanically taking certain actions. But it is helpful to understand that there are practices that can avoid violating the natural laws of development. The best way to learn this is to understand why certain practices would solve our challenges if they were done. We still have the challenge, of course, to get them implemented.
To properly decide on what actions to take you must understand what your challenges are and which actions have a positive effect on them. The following figure shows the challenges most organizations have. Eight of the challenges that are usually most critical are highlighted.
Most transformations start with some of these seven. But before deciding what to do it’s worth seeing all of the options available.
Print out the figure above and take a few minutes writing down which of the above challenges your organization is experiencing. Fill out the upper right box leaving the line blank if it’s not a problem, 1 if it’s a serious problem, 2 a moderate problem and 3 a minor problem. Then, go through each of the potential actions that follow and see which ones will help the most.
Potential actions to take
Most transformations involve a few of a subset of these. They are marked with an ‘*’.
Value Stream Wide
- Lean leadership and management
- Create visibility across the value stream of both the work and workflow *
Strategy and Portfolio
- Strategies with OKRs and Lean Portfolio Management
- Agile Budgeting
Product Management and Discovery Intake Process
- Use MBIs *
- Use Lean Product Management to Create an effective discovery intake process *
- Sequencing the work patterns group
Development Intake Process, Planning and Development
- Have an effective development intake process *
- Effective Program Formation
- Use ATDD to write stories *
- Effective Team Formation *
- Program Level Planning *
- Quality team process *
- Cross-team Collaboration and Integration *
- Collaborate With Shared Services
- Control interruptions
Technical Agility
Ops and Release
Lean Leadership and Management
Getting leadership and management involved is critical. Leadership’s involvement is mostly approving of requests made to make the organization more Agile. Management’s involvement is to create the environment within which people can work better.
Lean leadership and management is required to ensure that everyone in the organization agrees to abide by the guardrails. This sets the stage for all improvement. But in addition, Lean leadership and management help reduce the following challenges in different ways.
Ineffective budgeting can be improved by committing the organization to work on what is important to it and agreeing to organize the teams around the company’s products and services.
Shadow IT is not necessarily bad. However, it has to tie into the support systems of the organization.
A poor discovery intake process can only be improved with management requiring whatever is agreed to be followed.
Working on too many things requires managing the amount of work in process.
Insufficient specialized skills can be improved by adding more capacity to the skills needed and by ensuring that the demands on these people are managed properly.
No line of sight to strategies and OKRs makes it hard for teams to align. But to create this requires both a commitment to necessary tools as well as management creating a system that can create this line of sight.
Lack of visibility of workflows causes significant waste. Teams are sometimes concerned about having this for two different reasons. Sometimes they are concerned it will lead to micro-management. And sometimes they don’t want to create the habits and discipline this requires. Either way, management must get involved.
To improve the lack of coordination between teams requires clarity on what the organization considers important. This requires management helping the teams align around what’s most important. Otherwise, they will be pulled in different directions.
Integration errors are really teams getting out of sync with each other and not discovering it until they try to integrate. The longer it takes to integrate the more effort it will take to fix this. Management creating a bigger picture for teams to work in often goes a long way for integration taking place earlier than it would otherwise.
Problems discovered late is usually caused by not keeping the bigger picture in mind. Another job of management is to help create the templates and workflow that helps avoid this trap.
Ops blindsided and pulled in many directions occurs when priorities are not clear and visibility between develop and ops is not maintained.
Marketing needs left out can be avoided with the bigger picture view that management must provide.
Create visibility across the value stream of both the work and workflow
Ineffective budgeting can only be improved if you can see what is being worked on.
Shadow IT has two forms. The first is when business stakeholders start their own projects. The second is when requests are made directly to developers and no one else knows that the work they cause is taking place. Creating visibility exposes both.
A poor discovery intake process can only be improved when all of the work can be seen.
To avoid working on too many things you must be able to see what you are working on.
When there are certain insufficient specialized skills, kanban boards can be useful to manage the requests for them. Having an explicit workflow enables avoiding starting work that will require specialized skills when they are not available.
No line of sight to strategies and OKRs makes it difficult for people to make decisions in the context of the bigger picture. Creating visibility means having it be clear where all work originated from and why it is important.
Lack of visibility of workflows is fixed by having explicit agreements about how work is done.
Lack of coordination between teams can be improved when all of the teams can see what each team is working on.
Integration errors occur when teams have misunderstandings about who they are going to work together that aren’t found until integration. Creating visibility won’t avoid these, but can make integration take place more quickly. This would have them cause less damage.
If problems are discovered late, having visibility on what’s needed includes templates that include what to check for.
Ops blindsided and pulled in many directions happens both when they can’t see work coming their way and can’t prioritize the work they do get.
Marketing needs left out can be avoided with the same means of avoiding problems being discovered late and avoiding blindsiding ops.
Strategies, OKRs, and Lean Portfolio Management
Most companies have a fairly well-defined strategy but have other associated challenges, including:
- not having agreement across stakeholders as to what’s most important from an objective point of view
- not understanding what the company’s minimum business increments are
- not having a way to map the strategy to the development teams
Fixing these challenges is not usually very difficult if C-level people are involved. It’s about:
- mitigating risk
- having a plan they can count on
- knowing they will get value from their investments
- having a resilient system so they can get some degree of predictability
Solutions in this part of the value stream include:
- Use OKRs (objectives and key results) to use as the basis for your strategies, initiatives, and definition of business increments. This creates visibility on what the organization is working towards. Also, the OKRs can be weighted by importance and used to normalize MBIs/MVPs/MVRs across different business stakeholders.
- Work with product management when they define MBIs, MVPs, and MVRs.
- Plan quarterly. This helps to create smaller batches of work thereby enhancing the ability to pivot.
- Think in terms of long-lasting products and services not in terms of projects to allow for reasonably stable teams.
However, unless C-Level folks are driving things you can’t usually start here.
The diagram shows the effect of having strategies, OKRs and Lean Portfolio Management. Here’s how this helps the challenges mentioned:
Ineffective budgeting. Being clear on your strategies and objectives highlights that value is achieved not through development but through the entire value stream – realization being most important.
Work items not sequenced. The only effective way to sequence work items from different product managers / owners is to have a common set of objectives to measure against.
Unclear requirements. Having clear objectives and well-defined customers make clarifying requirements easier and more effective.
No line of sight to strategies and OKRs. It is impossible to have this if you don’t have clear strategies and OKRs.
Lack of coordination between teams. Lack of coordination between teams often comes from teams being pulled in different directions. Strategies and OKRs can help align the work of teams.
Problems discovered late and marketing needs left out. Strategies and OKRs help create the big picture which helps focus on the end result.
Use MBIs
Eli Goldratt (creator of Theory of Constraints) once said: “often reducing batch size is all it takes to bring a system back into control.” Using MBIs is often the most straightforward way to do this. If you only do one change to your process I’d suggest it be to use MBIs.
If the CTO/CIO and/or product managers are driving then you can start here even if C-Level folks aren’t.
Use Lean Product Management to Create a Well-Defined Discovery Intake Process
- Use MBIs, MVPs and MVRs. This creates smaller batches, which as Eli Goldratt says “are sometimes all it takes to bring a system back into control.”
- Create and manage a well-defined intake process. This enables using pull to manage work levels as well as create visibility on what is to be worked on.
- Create well-defined acceptance criteria for all backlog items
This is perhaps the most important item to attend to. A well-defined intake process can be used to avoid overloading the development group. It also puts some pressure on product management to select the most important items to be worked on.
It takes too long to get anything done. This is helped by controlling how much work is in process. A proper intake process also can load balance the teams better.
Ineffective budgeting. Having a clear intake process makes it easier to control the capacity programs and teams are given to different MBIs. Budgeting is made easier by having clarity on what is being proposed to work on.
Shadow IT. There are two kinds of ‘shadow IT.’ One is when business stakeholders have their own people and do their own development. This is not necessarily bad. It is mostly a problem when this work is improperly done and not able to be well handed off. Another is when work is done off the record. This causes many problems because when this happens it is not possible to manage work in process.
Chunks of work too big. An intake process is used to ensure items being worked on are of the proper size.
Poor intake process. Well of course.
Work items not sequenced. This is part of a quality intake process.
Working on too many things. One of the parts of a good intake process is to limit how many things get started.
Insufficient specialized skills. There will usually be people who are required for many different initiatives. By having a well-defined intake we can control which work items get started in a way that doesn’t overload these people.
Lack of coordination between teams. Controlling the amount of work should make coordination easier.
Integration errors. The intake process can adjust the workload so that teams can integrate better.
Ops blindsided and pulled in many directions. An intake process that uses MBIs will include the work for Ops. It also gives ops a way to decide what to work on.
Marketing needs left out. An intake process that uses MBIs will include what’s needed for marketing to do.
Have an Effective Development Intake Process
Effective Program Formation
- flow or incremental (and if incremental, the length of the increment)
- how to conduct the planning
Use ATDD to write stories
Writing small, well-defined stories is hard for most Agile teams. The fact that they are usually not taught this until the second or third round of training is one of the biggest mistakes I believe organizations make in moving to Agile methods. Regardless of who is driving, this should be in the first line of training.
Effective Team Formation
Cross-functional teams are almost always the best choice, but they are not always possible to form. Often a combination of almost cross-functional teams along with the use of kanban boards for those who have to share people with critical skills is required. Having effective teams is a critical aspect of Agile.
Program Level Planning
- Have planning events be primarily about collaboration and dependency management
- Have teams pull work from the backlogs created
Quality Team Process
- Use mob-programming when appropriate
- Use dynamic mobbing when appropriate (see this article on dynamic feature teams)
Cross-Team Collaboration And Integration
There are several ways teams should collaborate. SAFe has one, but there are several others. I’ve included a few in part V of this book.
There are several ways that teams can collaborate. A common one is to have a common cadence and integrate either continuously or no less frequent than at the end of the cadence. This is best used when teams are mostly using Scrum. It is possible to coordinate with a more pure flow model as well.
Collaborate With Shared Services
Control Interruptions
Automated Testing
Improving Developer Skills
Effective Architecture
Use DevOps Phase 1
The first Phase of DevOps is for the two groups to work together. This makes a surprisingly large difference and can be done before attempting to do Continuous Integration and Continuous Deployment (CICD).
- Ensure all work is visible to ops as soon as possible
- Include all tasks necessary to realize value when an item is released are included in MBIs, MVPs, and MVRs
Use DevOps Phase 2
Continuous Integration and Continuous Deployment (CICD) enables quick release and delivery.
DevOps is growing in popularity and for good reason. However, it is not much more than a specialized way to do Lean flow. Doing the same for the relationship between development and test is equally important.
Coordinate with marketing and support
Summary of Solutions to Challenges
The matrix below summarizes the solutions presented earlier in this chapter. A useful exercise is to print this out and fill in for your own organization.
