Transitioning to agile methods
While agile methods are not overly complex in and of themselves, it can often be very difficult to transition to agile, especially when many organizations are entrenched in traditional processes that are difficult to move away from. To some degree, we all fear change, especially when that change is uncertain and misunderstood. Many people have a deeply rooted mistrust of agile and lightweight methods possibly due to certain misconceptions about agile, hearing horror stories about agile, or even due to prior experiences with a process that was called agile, but in reality was far from it.
According to Forrester Research, “More than half of enterprises that aren't already using agile processes are interested in adopting them. But many of these shops are unclear about what agile adoption really entails.” (Schwaber, C., 2007, ¶1) This leaves many organizations unsure if moving to agile is the right approach for them and unclear about how they should move forward. Even though agile is becoming more and more mainstream and agile projects are continuing to prove successful, many groups are unsure how to effectively realize the full benefits of agile and are unclear what specific steps to take to even start their transition.
With this in mind, organizations transitioning to agile or scaling agile to a larger group of teams should start with a game plan that will address the potential challenges the organization will face and provide the necessary support if teams struggle with the transition. This paper will discuss how to build a good agile transition plan by outlining thirteen guidelines that will help organizations make a more successful agile adoption.
1. Start with a Plan
Organizations moving to agile methods should start their implementation efforts by establishing a clearly laid out plan of attack. Bring the right people together from within the organization to determine common objectives, goals and key milestones, as well as to think through a clear strategy for adoption. Too often, a team decides to go agile and fails to adequately inform and get input from all involved parties. There may be significant obstacles that may not be on the radar of the team that other groups are aware of. There may be policies, standards, constraints, and regulatory pressures that may impact how an agile adoption should be executed. A successful transition will include not only an organization's execution teams, but will impact the entire enterprise, including groups like portfolio management, enterprise architecture, operations, release management, and others. Expecting to become agile effectively while leaving out these key players can be a certain path to disaster.
With the appropriate contributing parties, the organization should then choose an adoption strategy that might be the best fit given their constraints. They may choose to move all teams to agile at once, start with just a few agile practices, and gradually add more as they mature, or start with a pilot team or project that can be fully supported and can choose some of the agile concepts the organization would like to employ. Regardless of the chosen strategy, leaders should fully understand what it is they are trying to accomplish and determine how they'll know if the strategy was successful.
2. Embrace Change
In order to truly transition to an adaptive way of thinking, things will most likely need to change. Don't resist, don't fight it — change is often necessary in order to gain any type of improvement, certainly the significant improvements that agile methods provide. There may be uncertainty, hesitation, and even skepticism, which is natural and even healthy, but the only way to truly understand if the benefits of agile can be realized is by giving agile processes and practices a try. Remember, change is only required because our current process isn't working as well as we'd like and trying new things is often the only way to get improvement.
In one organization, the teams were excited to move to agile and tried to convince their managers of the significant improvements they could make by doing so. Management's response was, “We don't mind if you move to agile, just don't change anything.” It takes much more than just lip service to make a transition successful, so hoping that nothing changes yet expecting improvement is just not going to fly. Keep in mind that one of the worst things that could happen by moving to agile is that we fail in our attempt. Certainly that could mean a significant cost, but at least we'll learn that agile doesn't work for us and we could always go back to what was not working previously.
Agile methods exist because traditional project management techniques are sometimes unable to keep up in today's ever changing environments. Rather than resist change, embrace it. Communicate why we need change and why now. Illustrate to decision makers the problems we are experiencing and how they are impacting our ability to deliver continuous value. No one wants to hear about solutions to problems they don't even know exist, so focus on what those problems are and then how agile might help to address those problems. Many will need to go through a complete paradigm shift as they learn iterative and incremental development strategies. This type of change is never easy, but when embraced and implemented the right way, it is certainly worth it.
3. Don't Force Agile
I had the opportunity to work with one large organization that was very motivated to transition to agile. Management knew of the tremendous opportunities and wanted to take immediate advantage of agile methods; their approach was to proceed right away and they embarked upon the huge effort of training the entire organization and standing up these newly formed agile teams. This in itself had some incredible risks, but to make matters worse, executive management informed all teams they would each be adopting Scrum, implementing two-week sprints, utilizing story points, and releasing software every month. Well, this sounds great…except that many in the organization had never heard the terms Scrum, Sprint, or Velocity and it seemed to them that management was forcing a new methodology down their throats. Additionally, these “mandates” were completely irrelevant to some teams that operated in a completely different manner than most, including teams that worked from a rapid queue. They couldn't plan for more than a day without getting change, let alone plan for two weeks. Others worked in areas such as infrastructure, data warehousing, and enterprise architecture, in which the exact same implementation as their application teams would be impossible.
Be mindful of the different situations your teams face and the complexities that may exist with their particular workload. A ‘one-size-fits-all’ solution for an organization rarely works well. Agile practices are meant to be tailored and adapted to account for a specific team's unique customers, risks, project limitations, and other challenges. For many teams, Kanban may be a better fit than Scrum; for others a blended approach may work best. The many agile frameworks are intended to allow organizations and teams to think about what would be the best fit for them and then continue to inspect and adapt as they become high performing.
It is also wise to not force a top-down transition approach. I always prefer an approach where there is plenty of support at the top, but lots of excitement and enthusiasm in the trenches so our transition can truly meet in the middle. This may mean starting slowly by encouraging individuals to attend lunch and learn sessions to become familiar with key agile concepts. Establish a forum where people can share experiences and offer contributions. Share very clearly and visibly the goals and objectives behind an agile adoption and ask for input on how to best approach the transition. You may want to run a pilot to allow those most enthusiastic to get started while allowing others to see from the outside how things are going. Be sure to share how things are progressing, communicate the pilot's results, and emphasize any successes. Getting buy-in at all levels may be the single biggest success factor to ensuring your agile transition will be successful.
4. Establish an Agile Working Environment
Sometimes a team might have full management support, complete team buy-in, a great strategy for adoption and yet fail miserably because they failed to establish an environment in which they could succeed. Agilists often speak of building small, co-located, self-organizing teams; yet, I'm always surprised to see an “agile” team of 15 or more people trying to work together daily; or a team in the same building, yet sitting in independent silos where they can't collaborate; or, even worse, a team that has great intentions and motivations, but is told by their manager how to do agile, which stifles their independent thinking.
Don't let the difficult endeavor of an agile implementation become exemplified by not providing an environment that can even support agile. Building cross-functional teams is a great start, but finding a way to allow them to sit near each other to foster communication and collaboration is important too. If the team size is over ten individuals, seriously consider splitting them into two sub-teams. They can still work together often and will most likely have some cross-pollination across the teams; it can be so beneficial to have a smaller group to talk through specific user stories, estimate work, sit through planning sessions, and run short daily standup meetings.
If at all possible, provide collaboration space for teams where they can come together frequently to discuss ideas and solutions, resolve impediments, and talk through their work. Given constraints on space, you may need to be creative and have teams share rooms, put up partitions, or even work off-site. If possible, provide space to build a physical task-board to help facilitate their discussions and make their work visible. You may hang whiteboards in the hallways, get roll-away boards, or even use a window. Allow team members to come up with ideas to make their space and collaboration tools reflective of their personalities.
If the team has distributed members, provide for them the tools they will need to communicate effectively. These may include video-conferencing capabilities or web cams, online meeting tools that allow shared screens and virtual whiteboards, a project management tool with a virtual task board, and so forth. We are so fortunate to live in an era where technology enables team members across the world to communicate easily and effectively — it would be a shame to let a team struggle when these options are so readily available.
5. Find a Champion or Sponsor
It is difficult enough for an organization to embark on an agile adoption, it so much more difficult when there is no agile champion or sponsor to lead the way. The agile champion becomes an advocate for new agile teams as he or she provides support, raises awareness, gets executive buy-in, and provides opportunities for continued learning and skill development. The champion helps to educate those new to agile or resistant to change and evangelizes the goals, objectives, and successes of the adoption. The champion should hold regular forums for ideas and input, establish communities of practices around specific agile topics, and warn of fail points and pitfalls to be avoided.
The champion should also consider building an agile champions group or team to help the adoption at multiple levels and create an organizational backlog of items the group can address. Ideally, this group would consist of IT managers, executives, and business owners that are empowered to make decisions and share a powerful and consistent message to those moving through the transition.
6. Top-level Leadership
Even with a good agile champion, it may be difficult to transition effectively if they are acting alone and don't have the full support of key decision makers. This can certainly be a tricky arena. Many champions will attempt to run a pilot with a couple of teams and hope their success will be proof enough to get executives and leaders bought-in to an agile adoption. This seems logical enough; yet, I've seen many pilots end in miserable failure because they can't run a true pilot without management support. “Sure you can run an agile pilot, but I still want to see your full requirements specifications, project plan, hourly project estimates, and complete status reports. Oh, and by the way, you still need to use our current project management tool and complete all the required project documentation!” Now the pilot team feels that they must do their traditional planning and documentation in addition to their agile planning, leading to redundancy and waste, and ultimately the team slows down, loses their enthusiasm, and abandons their agile practices.
When considering a move to agile, ensure you can give it a fair trial and have the right pieces in place to allow success. Forcing a pilot without executive buy-in can do tremendous damage as the pilot fails due to elements out of the team's control, but agile takes the blame. I always prefer to provide some education to those key decision makers and help them understand what we're getting into and how things might change. Encourage leaders to take the time to learn what an agile life cycle might look like and how they can support a trial. They may need to remove some processes, temporarily eliminate performance metrics, and provide the right environment in order to adequately evaluate how agile can work for us.
If you try moving to agile without knowing how others might respond to it, you run the risk of alienating the people whose support you need most in order to make an effective transition. Start by finding out how receptive others are to learning about agile and trying it out. Discuss with those who might be in a position to help or hinder your progress and collaborate with them on determining the appropriate steps to take. This will go a long way in your overall success, even if it's not as rapid an adoption as you would like.
7. Create an Agile PMO
Once you have executive buy-in and have given agile a fair trial, consider establishing an Agile Project Management Office (PMO). When included as part of an organization's agile adoption strategy, the PMO can help spread agile principles and practices across the organization, while ensuring there is some level of consistency in doing so. They can also assist to make sure individuals and teams have what they need to make the transition, the projects themselves are conducive to an agile approach, and the appropriate amount of process is built in to allow for an accelerated and adaptive delivery.
The agile PMO can support agile teams by establishing a good learning roadmap by both role and phase. Teams may need combined training to get started, specialized role training as they learn to execute and scale, and advanced training as they mature and contribute to an enterprise-wide transformation. Exhibit 1 is an example of a Learning Roadmap by role and phase of maturity.
Exhibit 1: Learning roadmap by role and phase.
The agile PMO also supports teams by providing coaching in which individuals with deep experience can individually work with teams or specific roles to help them understand and execute their responsibilities. The coach may participate in the project work with the team to show them the ropes and help them improve. The PMO can be instrumental in identifying and training internal coaches that can help with transitioning new teams and providing ongoing expertise.
From a project perspective, the agile PMO can lead the way by assisting with some of the project responsibilities and allowing new teams to focus on their agile adoption. These may include areas like helping with project reporting, establishing appropriate agile project metrics, assisting with any compliance-related work and documentation, and contributing in managing the flow of the portfolio of projects. This will ensure there is an appropriate amount of projects in progress at any one time to minimize project multitasking and context switching, which can be detrimental to new agile teams.
Finally, the Agile PMO can provide substantial value to an organization by establishing a clear and adaptive process framework that new teams can utilize as a guide in their adoption strategy. This may include providing guidance on the right tools teams can utilize, including agile project management tools, automation tools, continuous integration tools, and so forth. The PMO can provide guidance on establishing consistent practices teams can utilize across the organization without forcing specific methods. They can help teams eliminate waste by guiding them on streamlining documentation, artifacts, reviews, meetings, and handoffs. The PMO can establish communities of practices to engage individual contributors to help out in certain areas of the agile adoption. And, last of all, the PMO can be an exemplar to other teams by practicing agile themselves in their own activities and demonstrating a model of success.
8. Establish Communities of Practice
One activity I mentioned the agile PMO can help with, but is wise to implement regardless of who owns it, is establishing communities of practice (CoP). These communities of practice are generally led by the PMO or champions group members and are filled by individuals on agile teams who have a particular skillset or experience that would be useful in helping the larger organization. CoP team members typically spend a few hours each week on each iteration and helping to establish guidelines, best practices, and useful templates, or simply generate unique ideas around certain areas of an organization's agile adoption. These areas could be more general or very specific, depending on the needs of the organization. For example, communities of practice could be built around general areas such as Quality Practices, Release Planning, Iteration Execution, and the Product Backlog, or more specific areas such as Test Driven Development, Automated Testing, User Story Writing, Daily Standup Meetings, and so forth. Communities of Practice may also be created around specific project roles such as Scrum Masters, Product Owners, Quality Assurance, and Programming.
The objective behind the CoP is to bring together like-minded individuals working in the trenches to discuss and establish some agreed upon guidelines and best practices around their area of focus for the benefit of the rest of the organization. These CoP team members can then provide guidance and/or training via wiki sites, lunch and learns, team presentations, or even one-on-one with specific teams or individuals to help spread the knowledge and expertise. This not only helps new or struggling teams improve quickly, but also allows those with a passion around certain agile concepts to take a minor leadership role and provide value in a different yet very visible way.
9. Understand and Communicate Cultural Changes
Many organizations have operated in a way in which their culture embraces change, is very adaptive, and is always looking for ways to improve. If your organization fits this mold, I commend you. For those of us not quite there, a transition to agile may require significant cultural changes, which may be very difficult for some people to embrace. The first step to success is to recognize our deficiencies and understand what areas of our culture might need to be adjusted. For example, we may have a culture in which starting meetings late and not having clear meeting objectives is the norm. If we desire to simplify, eliminate waste, and increase productivity, our patterns for meetings and planning sessions will need to change and sometimes change dramatically.
Another example may be with organizations that have clearly operated in a top-down command and control environment. Teams take their marching orders from management, need approval to implement an idea, and go through multiple sign-offs in their project execution. Moving to a model of Self Organization, Empowerment, and Servant Leadership can be extremely difficult and will require a complete paradigm shift.
As we recognize areas in which our culture may need to change, we should also understand it won't happen overnight and a clear communication strategy will be necessary. We should show teams that we embrace the change and are taking actions to facilitate the change. It may be wise to hold one or several all-hands sessions to communicate these changes and outline a well thought-through roadmap to implement these changes. Allow teams to genuinely see that management is on board and not just giving the agile adoption lip service. Set realistic goals and visibly show the progress made on them. Recognize and reward good behavior. Encourage team building activities. Create a work environment that supports a healthy balance of work and relationship building. Empower teams to modify their own workspace to allow for optimal collaboration.
Take to heart the example of one very agile-minded company that demonstrated their agile culture in all aspects of their work. As this organization moved to a new campus across town, management simply identified where the teams would be located and then allowed them to self- organize. “Here are some tables, desks, chairs, wall partitions, and so forth. You have full access to our facilities group. You are empowered to design and create an optimal team workspace that will work best for you.” What a powerful message! Many teams did some very unique things that raised a few eyebrows, but it worked for them and that was the key.
10. Don't Throw Out the Baby With the Bath Water
Certainly some areas of our current culture, current process, and methodology will need to change to support a transition to agile, but be careful to not throw out the baby with the bath water. Yes, an agile delivery looks quite different compared to many traditional methods, but don't throw away everything you've learned about building your products. Many of the specific activities of developing software will not change much. You still need to talk to customers to know what to build, write code, create and execute tests, provide documentation, and so forth. Agile methods simply provide a different framework for those activities and provide more of a focus on team collaboration within each activity.
It is wise to start your agile adoption by taking a close look at our current processes and identify which ones have worked well for us, which ones might just need some improvement, and which ones we should completely abandon. Agile in no way, shape, or form tells us to throw out something that is necessary or is already working (yes, this includes documentation), it simply provides guidelines on how we can streamline, improve, and create more of a value-driven focus.
11. Allow Time to Improve
One of the most frequent fail points I see in organizations moving to Agile, is that they are way too aggressive and don't give themselves the necessary time to stabilize and improve. Yes, we've all heard about these miracle companies that achieve success overnight and within weeks have tripled their productivity. Sure, it is possible and it does happen, but don't expect it. In fact, you should probably expect to see a decrease in your productivity at first as the team tries to adjust and understand what to do. It takes time to go through that learning curve and organizations that expect immediate improvements are probably going to be disappointed. Don't give up. It may take several iterations, but if the team is truly inspecting and adapting through daily communication, retrospectives, and frequent collaboration with their customer, improvements will follow and the team will soon enough find they are in a better state than before the transition.
If the team or organization is just not making any improvement, seems stuck in their process, or has become frustrated with one or more practices; this may be the sure sign that an external coach may be necessary to provide a fresh perspective and provide recommendations for moving forward. It's very difficult to achieve success if we don't know what success looks like — a coach can help in those areas.
12. Retrospectives at Multiple Levels
Most agilists recognize that one of the most effective methods for building continuous improvement is by holding regular retrospectives. However, did you know that if you're not holding retrospectives at multiple levels within the team and organization, you are missing on significant opportunities to improve? Additionally, if you are not conducting retrospectives in an appropriate way you may be missing out on considerable growth. I'm always intrigued when an organization tells me they do a great job holding regular retrospectives but they never seem to get open and honest feedback from team members. Why is this? Often this is because managers and leaders are attending the retrospectives, which can cause team members to feel threatened or reserved, and certainly unwilling to talk about the challenges, problems, and even mistakes that were made. Running retrospectives at multiple levels will not only help with this, but provide forums for additional opportunities to get feedback.
Team Retrospective: The team retrospective is most common and should be done at the end of each iteration or, for non-iteration based teams, at intervals of 2 to 3 weeks. However, if these are to be of maximum value, the team must establish a safe, comfortable, and engaging environment. These should be held for core team members only, so functional managers, stakeholders, and executives should not attend this type of retrospective. Don't worry, you'll get your opportunity, but allow your teams to work through their performance impediments in a way that eliminates any kind of external pressures or threats. I recommend teams start with snacks, a quick team-building exercise, or some other fun activity that will allow them to feel at ease and openly share their thoughts. After the session, get permission from the team to share any non-sensitive outputs with others within the organization for continued learning. Remember, team retrospectives are focused on the team itself and their processes, not so much on the product, which is discussed during reviews. Any type of team, from an agile execution team to a product or executive team, can greatly benefit from regular team retrospectives.
Managers/Stakeholders Retrospective: Since we don't invite managers and stakeholders, I recommend holding every so often a retrospective specifically for them. You may choose to do this separately with managers and then with stakeholders, or all combined. This should be held by the team every 2 to 3 iterations. The objective is simple — after the team finishes their own retrospective, invite managers to join the team and provide a forum for them to communicate what they see is going well and what they see needs improvement. The outputs from these sessions are typically very different those the team comes up with, because managers and stakeholders have a completely different perspective and have the additional benefit of working with multiple agile teams. They can share any best practices they've seen, provide recommendations and, most importantly, show their appreciation to the team for the things that are working.
Project Retrospective: Even with the team running retrospectives every iteration, it is still very important to run occasional project retrospectives. This may naturally happen at the end of a project, but it is more valuable if they occur throughout the project so steps can be made to address any current issues and roadblocks. This retrospective should include the project teams, shared resources, managers, and any other interested parties. The focus of these retrospectives is more around the project itself, communication across these groups, and any ideas for improvement at the project level.
Organizational Retrospective: The organizational retrospective is held less frequently, possibly once every quarter or so, and is intended to bring the organization or department together to discuss how to improve organizationally. Because of the size of the group, I recommend asking folks ahead of time to create an individual or team list of items that should be discussed and then given to the organization so a game-plan can be created. This is a great way to encourage open feedback from anyone in the organization and can really show management's commitment to continuous improvement and the agile adoption. These sessions sometimes start with the larger group then move into breakout sessions to cover certain topics, certain teams or individuals can present, or management can intermingle with teams to address any areas of need.
13. Understand how Architecture Design and Requirements Definition Fit in an Agile Approach
One common misconception about an agile delivery process is that architecture design and requirements definition are only done within iterations and it is taboo to think about them too early on. Welcome to a world of rework and waste! Although it's true that a lot can be done iteratively and many details are intentionally deferred until we have better data, agile teams should definitely look at the bigger picture early on and build out high-level designs, requirements models, proofs of concept, and anything else that will be necessary to gain a shared understanding of the project. Agile project and release plans build into their timeline an Iteration 0 for these very activities.
Iteration 0 is so named because at the end of this unique iteration, the customer generally doesn't see any tangible value yet; however, no iteration is as valuable to the team as the Iteration 0. In this timeframe, the team sets up a foundation for success for future iterations. Take the necessary time to understand high-level objectives and fill out your product backlog and with the customer. We realize it is impossible to come up with all stories for a release or project, but this doesn't mean we don't put due diligence into making the backlog as complete as possible. The backlog helps to reconfirm the vision for the product and helps the team identify how they need to organize the work they are ultimately responsible for. Don't forget to bring the team together to create stories for foundational and purely technical elements that are necessary to create a good foundation for the project.
Bring in architects, database engineers, security experts, user-experience gurus, and other shared resources to collaborate on building an appropriate framework with the execution teams. This is the time to do some initial infrastructure and deployment planning, create high-level architecture designs, build out models, or potentially a proof of concept to explore any highly complex and risky elements of the project, as well as time for the team to continue to explore the backlog requirements and stories with the customer, elaborating when necessary. Keep in mind that any efforts during this time should be Just Good Enough so the team isn't taking precious time from getting to the actual execution iterations. Most of the details will be explored Just in Time when the team is preparing for and then executing iterations; so, up front we focus on what is absolutely necessary. The question to ask ourselves is: “Do we really need this right now, or can we defer it for later when we might have better information?” If you need something now, by all means do it; but if it can be deferred, it is generally better to wait until the Last Responsible Moment to allow some time to adequately think things through and gather the relevant data.
Transitioning to agile methods can be a challenging endeavor and it does require significant effort from many areas within the organization. There is no ‘one-size-fits-all’ approach to an agile adoption and it may take several attempts to get everything working seamlessly. By taking into consideration and implementing appropriately the thirteen guidelines described in this paper, organizations can more adequately overcome challenges, roadblocks, and areas of resistance; build support; and establish a realistic roadmap to a successful agile adoption.
Schwaber, C., Leganza, G., & D'Silva, D. (2007). The truth about agile processes: Frank answers to frequently asked questions. Retrieved from http://www.rallydev.com/sites/default/files/The_Truth_About_Agile_Processes_Forrester_white_paper.pdf
© 2012, Bryan Tew
Originally published as a part of 2012 PMI Global Congress Proceedings – Vancouver, British Columbia