Abstract
This paper describes the lessons learned and common problems encountered when introducing agile methods into organizations whose project managers traditionally use a PMI-based approach. It is intended to identify practices that work well and alert people about the common resistance areas to be mindful of. It reviews the “W5” (why, who, what, when, and where) of agile adoption and describes how most of the lessons learned and areas of resistance are people focused, not process focused.
Introduction
In 2004, the author presented the paper, “Utilizing Agile Principles Alongside the PMBOK® for Better Project Execution and Control in Software Projects” at the PMI® Global Congress (2004) conference in Anaheim, California, and was invited to teach a class on these new approaches. For the last five years, I have been teaching the popular two-day PMI SeminarsWorld® training course, “Agile Project Management.” This paper highlights some of the lessons learned from helping project managers adopt Agile methods, along with the common pitfalls and areas of resistance faced when making the switch.
Why Change?
First of all, with any change initiative we should clearly understand why change is necessary. If an organization is having success with their existing process (be it formal or chaotic), I would question the need to introduce a new methodology. Instead, perhaps our efforts could best be used by addressing incremental improvement. Switching to agile because your CIO read a magazine article on scrum or because it is “cool,” is not a good reason and a weak driver in pushing beyond the inevitable obstacles of a successful, complete implementation.
So, Lesson Learned #1 is: Before introducing agile, make sure there is a clearly understood, agreed on, and articulated need.
That said, most organizations have plenty of room for improvement in their software development process. The Standish Chaos Reports (2000–2009) repeatedly give low figures for project success rates and metrics, as follows:
• Only 16% of project are completed on time and on budget
• 31% of projects are cancelled
• 53% of projects’ cost growth exceeded 89%
Although the Standish Reports show small signs of improvement each year, the main theme is that the majority of companies continue to struggle at delivering successful projects. Given that one of the definitions of stupidity is “stupidity: doing the same thing over and over and expecting different results,” organizations need to look to different methods to address the process shortcomings.
Who to Change?
When planning the introduction of agile methods, we have far more than just the development team to consider. In fact, the successful adoption depends upon a variety of stakeholders, including:
Executives
- We need to win their support and buy-in to the change. They are our enablers and obstacle removers
Project Sponsors
- These people hold the budget and are powerful project influencers. If we want to get approval to try new approaches then we need to convince these people too.
Managers
- Managers are typically the resource controllers, so we really need them on board. They can be useful allies or roadblocks to adopting a new process, so make sure they are on your side.
The Development Team
- This is where the real change is made, and we need to make sure they are truly engaged in the process and involved in the planning and roll-out.
The User Community
- Often, the user communities are new partners in the process. If they have previously played only a passive role, we need to now ignite them with the opportunities for more control and work on new engagement models to get them involved.
Supporting Groups
- Specialized groups like a database group, quality or a project management office can be enablers or blockers to a process adoption. We need to ensure they have an input too.
It is not that you need to get formal written approval from each of these groups before introducing agile. Rather, be aware that the success of an agile introduction goes far beyond the development team. So, plan to interact, communicate, and win over as wide an audience as you can.
Lesson Learned #2: Make sure you engage all the necessary stakeholders.
What Changes Might be Necessary?
Just what may require changing is likely to be wider ranging than you might think, and these typically include:
New methods of decision making – Dictatorial, command-and-control decision making does not fit agile methods, and because the team will be doing more of their own local planning, we need to have better ways of gaining consensus and resolving conflicts.
Empowered teams – Are the project managers ready and willing to turn over iteration planning to the team? Is the team ready? Although iterative projects can be run as multiple, short waterfall projects, the real benefits of agile are only realized when we learn how to tap into the ownership, power, and accountability of empowered teams.
Business prioritized features – Is the business ready to get involved in prioritizing stories and evaluating increments of code on a regular basis? Is the project team ready to accept these development priorities and be directed by their customers?
Working with a new group of colleagues – For some organizations, having the development team work closely with the users of the system is new and disturbing. Some developers like stay in the background and code, deep in the moment of creative development. Users have a habit of interrupting coding nirvana with nasty exceptions to business processes and special cases that not all developers enjoy. Likewise, many business people equate working with the development team with having to join their mechanic in the muck and oil of a garage when getting their car repaired. It is an IT job, so do they need to be exposed to it?
Doing your job differently – A switch to agile is likely to lead to less project documentation and more change requests. (The two are not directly linked; we are not getting more change requests because things were not documented properly. Instead, the user community is engaged throughout the life cycle and given more opportunities to interact and think about the evolving system, so higher percentages of changes are uncovered during development when the costs of change are much lower than after deployment.) Also, agile change processes welcome late-breaking changes, whereas many traditional change management processes are really “change prevention” processes. However, some people will feel uncomfortable without the two-inch thick binder of (out of date) requirements sitting on their desks. Document-centric people will be forced to communicate more with other project stakeholders and attend software demonstrations to get the information they need, which, for some people is uncomfortable.
Relating to one’s superiors and/or staff members differently – The role of a project manager on an agile team is more a servant leadership role than a command-and-control directing role. Most good project managers already know that they are there to make the project team successful; they do not add a lot of business value to software products so they need to support the team. However, for some project managers, this downward serving model and increased levels of team planning and decision making can feel threatening. Make no mistake: there is still a lot to be done, but just the role and relationship may be different than previously experienced. Likewise, the team needs to be able to communicate bad news as well as good. Teams or cultures in which giving your boss bad news is difficult, will be a struggle in adopting agile methods. Project managers or scrum masters cannot remove impediments or help with resourcing problems if people are afraid to tell them what they are.
Lesson Learned #3: Consider the full scope of the changes required.
Lesson Learned #4: Consider the social factor impacts of the change.
When to Change?
We need to consider the timing of the change. For the introduction of agile methods, it may well be appropriate in the following situations:
When there are problems with current processes – if the current process consistently delivers late, over budget, or poorly accepted software.
When new projects occur with high levels of requirements uncertainty – for instance, a project with vague requirements. “We need a customer self-service portal” that is likely to have a high rate of requirements evolution as the details become clear.
When new projects occur with high technical risk – for instance, using new languages, unprecedented technology, and untested interfaces. Whenever there is high-technical risk, the iterative nature of agile projects is great for reducing risks early in the life cycle.
Time critical projects – for instance, if a product has to be ready for a November trade show, time boxed iterations and feature prioritization are effective ways of ensuring the best possible delivery for an immovable date.
Lesson Learned #5: Don’t assume it is always best to use an agile approach.
Where to Change First?
Deciding on where to first introduce agile methods can be tricky. It needs to be a real project; otherwise, people will dismiss any success as trivial. “Well, it was only a toy project; it would not work on a real world complex one.” Balance this with the need to see results quickly and roll out successful approaches to other projects, so they can reap the benefits too. So, do not choose a two-year plus monster project if you want to use the results to win over other groups quickly.
Real business need
• Not a toy project, but a real one people know needs to be done and is likely to be troubled by the same issues as other projects
High visibility
• One that people will see and have to take notice of
• Easy to publicize success (just make sure it is a success!)
• Has a good business spokesperson to spread the word
Three- to six-month schedule
• This is (just) big enough to be deemed a real project, but short enough to quickly make use of the benefits
With this approach, we run an initial project that has a real business need and is highly visible for 3 to 6 months. During this time, the team is training in agile methods and mentoring is provided. We review and document the project, publicizing its success and how happy the sponsor and users are. Then, individuals from that initial project can go on to be mentors for subsequent projects, scaling out the knowledge through the organization.
Note that I did not call the initial project a “pilot” project, because it was pointed out to me many years ago that using the term “pilot” indicates that something is a trial and may not continue. Obviously, our initial project is a trial, but why deliberately plant seeds of doubt in people’s minds? Call it an initial project, with the indication that there will be follow-up projects and you will avoid this uncertainty risk. Little items like these can make big differences on project introduction initiatives.
Lesson Learned #6: Choose an appropriately sized project to try the new approach on.
Lesson Learned #7: Avoid calling the initial project a “pilot” project.
“If you want to make enemies, try to change something.” – Woodrow Wilson
Recurring Resistance
As a believer in agile methods and someone who had witnessed the benefits they can bring and the great buzz of an effective agile team, I used to think rolling out the methods would be easy and simple.
Lesson Learned #8: Expect some resistance to the changes.
I was wrong; achieving successful lasting change is difficult. Changing processes is even harder, because a process is a system designed to resist change. Think about it: if every new type of requirement or defect that came along required a change to our development process, we would be in trouble. So, processes are abstract funneling techniques that were deliberately designed to resist change, which makes throwing them out or morphing them more difficult than changing, for instance, our time-recording system. In fact, there are a number of challenges, as follow:
Achieving lasting changes is difficult - Although people may be willing to try something new, it is a whole other story in getting them to switch to it completely. Many a promising start has reverted back to the old way of doing things because of this reluctance.
People are unlikely to blindly accept new approaches - People need to be convinced of the benefits of a new idea before they are willing to invest the effort in learning it. Many adults do very little learning because they just want to understand the easiest happy path through their regular work day and stick to that. For some people, asking them to think, learn, and adapt to new methods, in addition to doing their jobs, is like increasing their workload while keeping their pay the same—not that appealing.
Resistance to change is normal and healthy - We are bombarded by so many new ideas, goods, and services that it would be anarchy if we just adopted every new thing that came along. We need stability, standardization, and norms to function properly, which also serve as a filtering mechanism that saves people from having to think about every new thing. People believe that the good stuff will stick and the poor ideas will fade away. Unfortunately, some organizations are good at marketing poor ideas and some great ideas miss their mark, but on the whole, we get by.
So, we need to plan for resistance and have strategies in place to ensure worthwhile changes occur. Just explaining how agile methods are better is not going to ensure the whole scale rollout and adoption by an organization.
Lesson Learned #9: Don’t expect people to want to adopt agile methods because they are better.
Another issue is that the different stakeholder groups (executives, sponsors, managers, developers, users, and supporting groups) will have their own interests and concerns. We must sell the relevant strengths and address the likely concerns for each stakeholder group
Executives and Project Sponsors
• Interests: Economics, time-to-market, quality, competitive advantage, customer satisfaction
• Concerns: Risk of failure, unprecedented practices, counter-intuitive planning approaches
Managers
• Interests: Ability to cater to requirements change, risk mitigation, team morale, management load
• Concerns: Fear of a loss of control, fear of role erosion
The Development Team
• Interests: Effective working practices, meaningful work, work–life balance, less bureaucracy
• Concerns: Resistance to changes forced upon them by “management”
The User Community
• Interests: The features they want, ability to steer and influence project, better quality, visible progress
• Concerns: Fear of not getting all the required features, concerns around quality in early iterations
Supporting Groups
• Interests: Reassurance of process, clear communication channels, opportunities for intervention
• Concerns: Apparent lack of control, continual requests for involvement, lack of a visible end point
Lesson Learned #10: Understand the interests and concerns that apply to various stakeholder groups.
These stakeholder interests and concerns are of course a broad generalization; individuals within each of these groups will have their own interests and concerns. A key part of successfully engaging help in a change initiative comes from getting to know individual wants, needs, and concerns and then using these to help drive the change.
However, we need to recognize the hot buttons for the various stakeholder groups and learn how to effectively appeal to their wants. Equally, we need to listen and acknowledge people’s concerns; only then will they be willing to try something new.
Change Reaction
I think it is interesting that people often associate change with loss, and when we lose something we go through several recognizable stages of grieving. Do you find this hard to believe? Just read the following stages of loss and associated agile interpretations and see if you recognize any of them
Stage 1: DENIAL
Nice ideas, but they would not apply to my project, or how I work…
Stage 2: ANGER
No, I won’t change; you cannot make me use stupid user stories…
Stage 3: BARGAINING
Okay, how about if I have 6-month iterations?
Stage 4: DEPRESSION
This is not working and nothing makes sense…
Stage 5: ACCEPTANCE
It was difficult, but I can now see how it helps….
Lesson Learned #11: Don’t assume that people will act rationally
Triggers for Change Resistance
The sense of loss is a powerful reason why people often resist change. Fortunately, there is a wealth of research on understanding and effecting change. Books, such as How to Manage Change Effectively (1985) by Donald Kirkpatrick and Leading Change (1996) by John Kotter, list a number of circumstances in which people resist change. Anyone who is trying to roll out agile methods would do well to be aware of them:
People will resist changes when any of the following feelings are present:
- There is a sense of loss: be it security, pride or satisfaction, freedom, responsibility, authority, good working conditions, and/or status
- The change initiative and its implications are misunderstood
- Belief that the change does not make sense for the organization
- A low tolerance for change in our lives—perhaps there are already many changes going on at home
- When change violates a principle or commitment that the organization must stand by; for instance, customer service or quality
- If there is a lack of respect for those initiating the change
- When people are excluded from the change initiative—this a great way to generate resistance
- Changes viewed as criticism of how things were done in the past: “You are so bad at development we are going to have to adopt agile methods!”
- The change effort occurs at a bad time, or other issues or problems are also being handled; for example, just before another company take over
Levels of Recurring Resistance
When people oppose change, their resistance tends to fall into three categories:
Level 1 Resistance – Confusion (lack of information)
Level 1 resistance is the least severe and usually caused by confusion and a lack of information. We can tackle this by providing information in the forms of newsletters, lunch and learns, discussion forums, websites, and providing plenty of opportunities for feedback.
Level 2 Resistance – Resistance to change (fear of loss)
Level 2 resistance is more difficult to overcome because it is directed at the change itself and usually stems from a fear of loss of something valued. To tackle this we need to:
- Build strong working relationships with those creating the resistance
- Embrace those behind the resistance and get them on board as the official skeptics
- Listen with an open mind and provide acknowledgment
- Stay calm to stay engaged and maintain a clear focus
Level 3 – Entrenched resistance (beyond the changes at hand)
Level 3 resistance is the hardest to overcome because it is personal! Level 3 resistance has little to do with the change at hand and more to do with the people behind the change, old personal agendas, vendettas, and so forth. Fortunately, level 3 resistance is quite rare, but if you run into it you should be prepared for a long struggle. To help overcome it:
- Continually work on building relationships
- Begin small—find things you can agree on and proceed from there
- Candid conversation is vital—confront the issues
- Get people deeply involved in changes that affect them
- Support yourself and be prepared for setbacks; you will not be able to convert everyone, look after your health.
Lesson Learned #12: Expecting everyone to play nice.
Well, this all seems like doom and gloom. If people resist change under these circumstances and to these varying degrees, just when do they accept change? Fortunately, there are many circumstances in which people are supportive of change and by knowing about them we can try to ensure they are present and stack the deck in our favor.
Triggers for Change Acceptance
People will accept changes when the following feelings are present:
- When the change is seen as a personal gain: in security, money, authority, status or prestige, responsibility, working conditions, achievement
- Provides a new challenge and reduces boredom – when we create more interesting work
- Opportunities to influence the change initiative – when we involve people in the changes
- Timing – the time is right for organizational change
- Source of the change initiative is liked and respected
- The approach of the change and how it is implemented appeal to us – when they buy in to the approach being taken
These are the conditions we need to establish in order to stand a chance of having our change accepted. It is interesting to note that they are all personal, human-centered circumstances. There are no “When the change is awesome,” “When the change will reduce production costs by 60%” type conditions; yet, these are the types of drivers people try to use to introduce agile methods and then fail.
Lesson Learned #13: Make sure to address the personal side of the change initiative.
As a logical, rational person it pains me to acknowledge it, but as with making purchasing decisions, we make up our minds in our hearts first and then rationalize it with our brains. The same goes for accepting changes—we must first go for the heart by crafting a caring, inclusive, compassionate change environment and then follow it up with some sensible changes.
(Weird cults and religions learned this years ago: if you are nice to people and actually pay attention to them providing them all the attention they feel they have been missing, (sometimes called “love bombing”) they will be grateful and a certain percentage will even give you all their money; however, with cults, most people later realize they’ve been fooled)
I am not suggesting we use “love-bombing” or subliminal messages to introduce agile methods. We do, however, need to understand that a key component for change acceptance is the human side of introducing the change. The best changes in the world will meet resistance and take longer to accomplish if they are implemented without regard to the human side of change.
A Roadmap for Effective Organizational Change
Therefore, we need to consider both strategic and human-related steps and follow a proven route to effective change.
Exhibit 1 – A Balanced Strategic and Human-based Change Roadmap
The change roadmap shown above (Exhibit 1) uses parallel strategic and human threads to achieve successful change.
1. Establish the need – gain consensus on why a change is needed. Qualify and assess the organization and analyze and document current problems and shortcomings. Capture previous stakeholder complaints and issue log and postmortems problems. Keep it real, but if there is a burning platform from which we have to move forward, document it fairly. Determine the business benefits and describe where we are now.
2. Create a vision – describe a better state by outlining the goals and objectives we are aiming to create. Unite everyone with a common end goal of what success would look like. Describe where we want to be.
3. Form a change coalition – identify the key stakeholders and get people involved, not only on the initial project, but also on advisory and review boards. Provide mechanisms for general input and information exchange. Use websites, lunch and learns, and other forms of communication. Be civil, humble, and nice. Do not assume or give the impression the change team has all the answers. Ask people: How should we get there?
4. Communicate the vision – provide a clear outline of what is going to happen. People generally need to hear things five times, in five different ways, to ensure they stick; use different formats, analogies, and styles. It is generally impossible to over communicate a change initiative vision. Plan and promote the organizational changes.
5. Encourage employee participation – After inviting people to be involved, make sure they are. Schedule follow-up sessions and speak with people about their concerns. Ask for volunteer reviewers and give praise and thanks for reviews we get back, especially if negative. This is the opportunity to turn people around to your side, while the resistance is only at levels one and two. Work on forming good relationships and select and train the team.
6. Plan for and create short-term wins – identify the initial project. Schedule some early, small victories to build momentum, demonstrate progress, and reassure sponsors. People only trust for so long; give them something to justify their continued support.
7. Provide rewards and incentives – change, on top of a regular job, is a lot of extra work. Reward contributions as well as organizational norms will allow. If you cannot give bonuses, order high-quality food for the lunch and learns. Give away good mementos and freebies and arrange for time off if teams work hard on the initial projects. People have to see the benefits of taking part, otherwise they will not bother. Goodwill, loyalty, and corporate benefits do not work with everyone.
8. Consolidate improvements – Make sure the successful changes get repeated. Document the successes and spread the word. Monitor and perform mid-project retrospectives.
9. Institutionalize new approaches - Complete and review the initial project. Measure and promote business benefits and get the sponsors and users to promote the benefits. Identify the next project and broader roll-out plan. Make the changes stick by institutionalizing them by making them parts of the standards and culture. Support other groups trying to repeat the process.
Conclusion
As a project management consultant, I only get to help a couple of companies each year adopt agile practices. Nonetheless, I am pleased to say that, despite the mistakes I’ve made in the past, none of the over thirty companies I’ve worked with over the last 16 years has switched back to their original approaches after trying agile methods.
So, take heart, and plan your agile introductions in close collaboration with all the stakeholders who will be impacted. Listen, pay attention to the skeptics, avoid the common mistakes, and have faith that the good changes will prevail.