Transitioning to agile

Abstract

With the rising popularity and success of Agile methods in technology circles, many organizations have attempted to transition their development processes to a leaner, more customer focused development model. Why do some organizations gracefully transition to Agile methods while others do not? What are the common traits of success? What are warning signs of failure?

I have identified four key elements common to teams that successfully transition — traits that were absent in teams that fail:

1. Ensure Sponsorship.
2. Be Committed.
3. Find the Right Proj ect.
4. See Things Through.

Whether you are a manager driving a top down adoption of Agile, a team that has tried an Agile practice and failed, or an individual trying to drive adoption from the bottom up, you will find value in this paper.

Introduction

My move to Agile started like many others: as a response to failure. I was the program manager for a large infrastructure project that supported millions of end-user Internet subscribers. We had thirteen full-time people and a development lifecycle of sixteen months. Because the project had been successful to date and tracking to plan with no surprises for twelve months, I communicated to our executive management that our business would be able to take advantage of a financial incentive that was being offered by one of our network partners. The forecasted savings was $4m - $6m for a single quarter.

In month thirteen, as we exited the code complete milestone, we began the testing and stabilization phases of the project. It was then, for the first time, that we ran a realistic stress and load test, one that replicated actual usage patterns over the past year.

We started the test run on a Friday and reviewed the results the following Monday. We gazed at the results with a mixture of confusion and uncertainty. The results showed an average system response time of 5 seconds, much slower than our maximum threshold of 200ms.

After consulting with another team in our company, which was more familiar with the technology we were implementing, we realized that our design was fatally flawed – the technical implementation we had architected would never work in reality, at least not with the software we had available to us at the time. We re-architected our failed design and ultimately shipped our system, but it was three months late, and over budget and we all felt awful.

This was the catalyst of change for me. I had slipped projects in the past, but no single previous project had such a financial impact on project cost (thirteen people for sixteen months) and lost opportunity cost ($4m - $6m), not to mention the blow to team morale. I was determined to find a new way to build software, one that would mitigate as many project risks as possible while building a high performing team.

I spent the following three years experimenting with Agile software development principles and practices, and established a team that wanted to experiment with me. We excelled at the projects we delivered. During the second and third years of these “experiments,” I began coaching other teams, taking what we had learned and evolving other teams to become more Agile.

In working with these teams, as well as my own, I discovered four traits apparent in organizations successfully adopting Agile and absent in those that failed. These traits apply to all levels of the organization, from the main line development staff to the organizational executive management.

1. Ensure Sponsorship.
2. Be Committed.
3. Find the Right Project.
4. See Things Through.

Ensure Sponsorship

Sponsorship is more than providing funds and motivating people to “get the job done.” These elements alone will not provide the structure needed to transition to Agile. One executive manager I worked with was a true sponsor. His leadership allowed me, the team, and ultimately the organization to succeed in its adoption of Agile. His secret? Reward for failing early, provide protection and promote transparency, and accept the cost of transition.

I began to look for these elements in the executive management staff of the teams I coach. I noticed that, often, when one of these was not present, the team would tailspin out of control. These managers did not understand their role in the transition. They didn’t realize that for Agile to work they would have to change their own behavior or working style.

Reward for Failing Early

One of the biggest fears people have when joining an Agile team is the fear of failure. Successful people have a set of approaches, a set of tools that they use. These tools work for them. When asked to do something that is outside this toolset, outside of their comfort zone, they begin to shut down. In defense, they begin to poke holes and identify potential areas of weakness on why and how Agile will fail.

• Our organization is unique; it can’t possibly work here.
• Our code base is too complex.
• We need to document our work and Agile means no documentation.

The list goes on and on. What I discovered is that these are all surface issues. These are excuses that hide the truth, and the truth is that people are afraid of failure. No one likes to fail, and if one is asked to work outside his comfort zone, with an unknown set of tools, and is still expected to deliver at the same level of quality and predictability, he will fail.

Consider the following. If a person tries something new and fails, he likely will be punished for that failure. Consequently, he learns quickly to only do work that he knows he will be good at so that he will be rewarded for the higher probability of successful delivery (see Exhibit 1). When I work with teams that are adopting Agile, I propose the opposite. Reward people and teams when they take risks and fail.

Exhibit 1 – Punishing for Failure

Anytime we learn something new, we make mistakes. Recall your first time riding a bicycle, you were expected to fall. This changed over time. With practice, support, and encouragement we became better. How sad it would be if when we fell, we were instead punished for falling and told, “Don’t try that again.” Yet, too often, that’s exactly how we treat people in a work environment when they dare to learn a new skill, one that will ultimately improve their own toolset and the organization’s success: “That didn’t work. Don’t try it again.”

Management always talks about increasing productivity without adding employees. If they were to encourage and support people to try new things, they would find that their operational efficiency increases while their costs remain the same. This requires time and patience, but taking risks ultimately yields a higher benefit than always doing what you already know (see Exhibit 2).

Exhibit 2 – Rewarding for Failure

Sponsoring a transition to Agile means rewarding risk and failure, allowing individuals to grow, learn, and develop a new set of tools for their tool belt. After time, people and teams will become more comfortable with the new set of tools and will begin instructing and coaching others who are just learning.

Provide Protection and Promote Transparency

Providing protection for one’s organization and promoting transparency are tightly coupled. While all managers must do this to some extent, executive managers who sponsor the transition to Agile must be vigilant in this area if the team is to succeed.

As an example, in general, Agile projects are more likely to have cross-functional role boundaries. One member of a team I coached was Bill, a functional tester who also had the ability to write mainline code. When the test manager would ask what Bill was doing, we (the team and I) would respond “he’s doing work on the project.” This did not go over well. In this organization, people were rewarded for their contribution to projects based on their functional roles, not on their overall contribution to the success of the project. We identified this problem in the company reward structure, one that was impeding the transition to Agile, and brought it to the attention of the executive manager (and Agile sponsor).

He was able to rework the reward structure for the team. Everyone in the organization was able to see the new reward structure and how it motivated the right behavior, which was for the team to deliver a successful project to the end customers. The manager’s response not only helped team members and other managers feel better about cross-functional roles, but also provided proof that being transparent about problems and obstacles was a good thing.

When a sister organization began to get wind of what had happened, they started to raise issues. The executive manager was able to protect the team from the distraction, which allowed the team to deliver successfully.

Accept the Transition Cost

I am asked at the beginning of every coaching engagement or new-to-Agile team kickoff meeting if the team will begin to see the benefits of Agile principles and practices on day one. This question often comes from a senior manager in the company. My response is simple. The transition takes time and time costs money. If a team is currently able to produce ninety units of work per month with its current process, it will probably take three months for the team to get back to that level. During that time, team productivity may drop to forty-five units of work output or less (see Exhibit 3).

Exhibit 3 – Cost of Change

In every team I have worked on or with, that has been new to Agile, the breakeven point was hit between the end of the third month and middle of the fourth month of working together. Several factors contribute to this temporary drop in productivity. First, the team is trying something completely new. Second, if the team has been working together for a long time, they will need to re-learn how to work together. Teams, and the company, will go through normal development activities such as the forming, storming, norming and performing phases as documented by Tuckman (Tuckman, 1965).

Accepting this cost, and allowing time for the transition to occur cannot be cut. The longer the time to reach the breakeven point, the higher the cost incurred from decreased productivity. Creating an environment that allows people the ability to experiment and grow will condense the transition cost and allow the company to reach benefit sooner.

Be Committed

Being disciplined is hard. Ask yourself, how challenging is it to go to the gym on a regular, consistent basis? How challenging is it to stick to a healthy diet? How hard is it to maintain a consistent level of patience with a young child after he/she has done something bad? It’s equally hard to truly commit to all of the changes a transition to Agile requires.

I overheard a conversation recently in which one person appeared very passionate in debunking Agile and another was countering his argument. The conversation went like this:

“Why is Agile not in widespread use at the company? I mean, if Agile is really that good, you would think that it would be in widespread use at our company, but it’s not. We hire very smart people, and if the majority of these people can’t or won’t use it, what does that say about the methodology or the people? XP is not that new, it’s been around for years. And someone pointed out to me that the C3 project which gave rise to XP failed.”

The conversation continued, this time with the Agile proponent speaking up.

“I have an idea, let’s change what you just said and apply it to eating healthy and exercising.

Why is eating healthy and exercising not in widespread use? I mean, if eating healthy and exercising is really that good, you would think that it would be in widespread use at our company, but it’s not. We hire very smart people, and if the majority of these people can’t or won’t eat healthy and exercise, what does that say about eating healthy and exercising or the people? Eating healthy and exercising is not that new, it’s been around for years. And someone pointed out to me that the original attempts to eat healthy and exercise failed.”

Anything and everything can be shot down when people are not committed to what they are doing. Similarly, teams that merely practice Agile fail time and time again, often because they are trying to apply the knowledge learned in their legacy process to fit to a new lightweight method.

I’ve found that teams that were successful in Agile adoption embraced its principles and threw out the learned practices they had been using for so long. These teams moved to a collocated workspace so they could have a higher degree of interaction. They acted as a team, working towards a common goal by delivering and demonstrating working software on a monthly (or shorter) basis. They do not stop working because there is no more work to do in their functional area, everyone contributes. They worked hand-in-hand with their customers and stakeholders, and embraced change, understanding that it was impossible to capture all possible requirements during the traditional planning phase of a project (Robertson, 2006). Instead of looking for a practice to save their failing project, these teams looked to the principles to become the embodiment of the team.

On the other hand, teams that failed to adopt Agile principles and practices were never able to successfully change their mindset. They constantly contradicted the new process they were using and sabotaged the team and organization in which they worked. If they were adopting Scrum, for example they might not have a product backlog or skipped the daily standup meeting. They might extend the length of the Sprint (from four weeks to as large as twelve weeks) to fit the feature work they were being asked to develop. Team members were still tracked on an individual contributor basis and not on overall team performance and capacity. Teams did not (and were not encouraged to) hold frequent retrospective meetings. They never truly committed to the new methodology and, not surprisingly, they failed.

I knew a developer who was assigned to a project team that was doing Scrum and XP. He did not join on his own accord. He continuously criticized the way the software was built and frequently asked to work in his cubicle, outside of the team space, for two weeks and re-emerge with lines of code to integrate into the branch. Further, he did not want to participate in the team retrospectives. He saw them as a waste of time. This person reduced the teams’ output and after only a month almost caused the team to implode. He had no desire for self improvement, nor did he have any desire to have his worked critiqued by his peers. He did not look for areas to improve his skills and competencies. He was not ready to change.

It’s not enough for your sponsor, your manager, and your team to be committed. Each individual must also be committed to experiment, learn, question, and change his or her own behavior.

Find the Right Project

Of the teams I worked with, those that successfully transitioned to Agile were able to select the project that would best suit an Agile methodology. Teams that had projects thrust upon them, and who then tried to experiment with Agile principles and practices, failed the majority of the time. The successful projects had the following characteristics:

• Low-to-medium risk.
• Flexible deadlines.
• Relatively well known technology.
• An ability to adhere to the triple constraint.

Low-to-Medium Risk

Project risks come in many shapes and sizes. The most common project risk patterns I identified were technical risks, management and organizational risks, personnel risks, and schedule (time) risks. Teams and organizations that were able to mitigate as many of these as possible were often able to successfully transition to Agile principles and practices.

Technical risks are typically lower if the project, code, and architecture are new and the external interfaces are few. Technical risks increase when technology is not well known or if the project is taking on legacy code. Technical risks are also affected by the number of external interfaces and the history of the organizations that support those external interfaces.

Management and organizational risks are dependent on the flexibility of the executive and middle management team in the organization. I found organizations that have traditional command and control type managers often fail at adopting Agile methodologies. One causal factor is usually that the organizational reward system is not aligned to a team-based reward model. As a result, functional managers heavily influence their direct reports who work on Agile projects to only do the work that is in their functional disciplines. The functional manager has to represent his direct reports in organizational performance meetings and cannot accurately represent their contributions if he does not know what each person is working on, with regards to that person’s functional role. Without management or organizational support, people working on Agile teams are often overlooked when it comes to bonuses and raises, regardless of the success of the project.

Personnel risks focus on the individual mindset and skill set of team members. These risks increase when the people are subject matter experts and are required to split their time across multiple projects, do not have the skills required to implement the technical solution, do not have the will or desire to work in a cross-functional, collocated environment, or are split across geographical regions (anything outside a collocated work space can be considered a split). Personnel risks are reduced when people are willing and able to take on a team mentality and let go of the “I” success factor. The teams I worked with that were most successful were able to cohesively work together, break the cross-functional barrier, and focus on customer/stakeholder satisfaction by managing the end-to-end flow of the requirements instead of solely focusing on the functional area of the project where each individual is most competent.

Schedule risks involve the end delivery of the project and the dependencies the project has on other teams. If the project has a large number of external interfaces, intense choreography is often required to pull everything together. I found that projects that were able to stand alone, with little external dependencies, were more successful in adopting Agile than projects that had a high level of external dependencies and commitments.

Flexible Deadlines

I have worked on, and coached, teams that had both fixed and flexible deadlines. When either type of team transitions to Agile, they go through a period of lowered productivity, mainly because they are learning a new way to work; one that often crosses functional boundaries. The teams that had flexible deadlines flourished because there was enough slack in the system to allow them to go through team development phases, such as the forming, storming, norming and performing phases described above. Teams that had fixed deadlines often fell back into their old behavior patterns because they had constraints and expectations put upon them to not only change, but to deliver a software product by a firm deadline. The right project will factor in flexible deadlines.

Technology is Relatively Well Known

Another team in my organization attempted to try Agile practices and implement a new technology that was just coming of age. This was a complete disaster. The team was willing to try Agile; they had signed up for it. However, since the technology was so new, the team was not able to plan with any predictability. The project manager fell back into a traditional command and control style, because that is what she knew best, and the team began to spiral out of control. Team morale sank, people became frustrated and terse with each other, and everyone fell back to a cover-your-caboose type mentality. When transitioning to Agile, pick a project where everyone is familiar with the technology.

The Triple Constraint

A Guide to the Project Management Body of Knowledge (PMBOK® Guide) identifies the triple constraint as project scope, time, and cost (Project Management Institute, 2004). In Agile iterations, time and cost is fixed. Features are adjusted to fit within the iteration length (see Exhibit 4).

I often encounter the problem where people think they must have every feature in a product or service before it can be released. Based on my experience in software, not every feature is warranted. Sometimes people will request features in a product because without it, they would be out of a job. Unfortunately, the established mindset is that all feature sets, large and small, must be included in a release. As a result, all features carry equal weight. This often results in software where 20 percent of the features are used 80 percent of the time. Put another way, 80 percent of the features teams spend time and money delivering are used only rarely.

By fixing time and cost, estimating features, and then constantly reprioritizing those features to meet the needs of the business as the environment changes, teams are able to build and release the core set (that magic 20 percent) of features first. Teams can then re-evaluate the remaining features (the other 80 percent) to decide if they are truly necessary to deliver. If they are deemed necessary, they can be shipped at a later date as a point release upgrade.

Exhibit 4 – Triple Constraint Comparison

See Things Through

Based on my observation of several teams, the following seems to be one of the best ways to transition to Agile:

1. Start using Agile principles and practices on a pilot project with loose dependencies and commitments.
2. Pick a practice or practices and do them by the book for at least three months.
3. Inspect and adapt on the practices and use the team retrospective to improve team performance.

The teams and organizations that I have seen to be the most successful with their transition to Agile started with a pilot project. The project was made up of a team of volunteers who were willing not only to deliver the project, but also to experiment with techniques outside their skill and competency sets. These teams would document and report back to management on the success of the team and how its adoption of Agile was coming along. This had several benefits, chief among them being that issues were raised when they were small and consolidated.

A point of failure I see on a regular basis is when teams decide to modify a framework, say Scrum, at the beginning of a project, without any experience using the framework. I had a conversation with a team I was coaching on Scrum. After a thirty-minute conversation, I discovered this organization laid out Sprints in a waterfall type fashion.

• Week 1: Planning
• Weeks 2 – 5: The actual Sprint. During this time, functional developers coded and testers wrote test cases with very light testing
• Weeks 6 – 7: Integration and testing
• Week 8: Live site deployment
• Week 9: Buffer, just in case.

The team, in this case, was ultimately unsuccessful in their development and release of software. They were stuck in a traditional serial development mindset and as a result, only frustrated their management team on a more frequent basis. The teams most successful adopting the frameworks do them “by the book” for at least three months.

Successful teams use the sprint/iteration retrospective to fine tune the processes. The retrospective is not a time to review project status; it is a time to review team health. While this may sound like something that can be removed from the process, it cannot. On my current team, we missed two retrospectives—we went a full month before having one. During this time, events occurred with our client that caused us to shift how we addressed our requirements, and it was a shift in the wrong direction. In our retrospective, we all acknowledged that the shift was wrong; however, we did not use the mechanism in place to address it. As a result, a bad behavior was born and practiced by the team. We began to silo our work, creating a team of individual experts who did not have the insight or skill to work on other components of the system. When this was identified in the retrospective, we (the team) put a series of checks in place to undo this behavior. We instituted a rule in that the person least qualified, within reason, had to do that work, and not the expert. The experts were available for consultation and pair programming as needed by the other team members.

Final Words

Whether it’s changing your diet or changing your software development style, creating new habits is never easy. If steps are taken to ensure sponsorship, a fully committed and dedicated team is ready for the challenge and has chosen a project that fits, and the chosen framework (and only that framework) is adhered to for at least three months, there is a high probability of success in your transition to Agile.