Introduction
At the PMI 2001 International Symposium I presented a visual model that depicted why iterative processes are hard to manage using traditional techniques. This model, called The Blur Box, is part of a burgeoning movement within the software development industry that prefers agile, adaptive, and/or lightweight processes over predictive, inflexible, heavyweight processes. (Hereafter I will use the general terms agile/adaptive and predictive.) This movement, spearheaded by the Agile Alliance, is a reaction to the unnecessary overhead generated by many methodologies and business practices.
References to the Agile Alliance and other adaptive schools of thought are now everywhere. Project managers are discussing Extreme Programming (XP), Agile Modeling (AM), the Rational Unified Process (RUP), and other agile approaches with relief, acceptance, skepticism, apprehension, or derision. To reduce the viewpoints to the crudest possible caricature, see the table in Exhibit 1.
If you have read this far you clearly belong in the top row of Exhibit 1. The reality is that great thought must be put into the decision of whether or not to adopt agile processes. This paper addresses some of the key points for successfully integrating an adaptive approach into IT development. Along the way, quotes from some excellent new books/websites will point you to further sources.
A Summary of the Blur Box
The Blur Box model (Exhibit 2) depicts software development as a fuzzy blur of activity that is difficult to pin down or control. Development is organic and dynamic, and should inherently adjust to new discoveries, requirements, prioritization, or setbacks. Many of the management practices currently in vogue are based on the idea that it is good to predict where things are going in both the short and long term. The Blur Box reveals that we should have a general plan of where we are going, but no idea of how we will actually get there. The paths you will take will appear in front of you: don't peer too far into the blur ahead.
Exhibit 1. Agile Viewpoint Stereotypes
Exhibit 2. The Blur Box
Exhibit 3. Phases of the Blur Box
Exhibit 4. Thread Dependencies
Software development consists of four stages: Analysis, General Design, Detailed Design, and Development (Exhibit 3). Threads of work are concerted efforts that are somewhat autonomous; common examples of threads are Business Process Redesign, Documentation, Database Design, Software Development, etc. Threads wind their way through the four stages as needed in an unpredictable pattern (Exhibit 4).
For example, a team may briefly analyze, decide to write a prototype, and jump right into development. Upon the discovery of new questions, the team or individual members of the team may revisit Analysis or General Design. The point is the team gets to decide how to implement the requirements specified. Their goal is neither to follow a plan, nor make progress easy for you to report. Their goal is to deliver working software with as much simplicity and economy as possible. This makes life more difficult if your company tracks progress through Gantt charts and other predictive tools.
One implication of the Blur Box is that predictive tools and approaches are not successful. Instead of planning a year or more at a time, only plan one to three months ahead and be quite general regarding anything thereafter. A new book called A Practical Guide to eXtreme Programming by Astels, Miller, and Novak says it best:
“You must understand one thing about your plan (and there's no escaping this). It is inaccurate, and it will change as you learn more, as requirements change, and as new risks are discovered. In fact, the farther in the future that something is planned the more likely it is to be inaccurate and the greater the inaccuracy is likely to be. It is simply too difficult to predict too far in the future. By the time you get there, chances are, your plan will be totally unfounded in reality.” (p. 5)
This concept of not doing an up-front plan may give the impression that agile approaches entail less effort. This is not the case at all. Iterative processes have some downsides, and can provide several traps for the unwary manager or architect. These traps aren't necessarily worse than those experienced in other methodologies, but they are different from the traditional traps:
“Iterative development actually involves much more planning and is therefore likely to put more burden on the project manager: An overall plan has to be developed, and detailed plans will in turn be developed for each iteration. It also involves continuous negotiation of tradeoffs between the problem, the solution, and the plan. More architectural planning will also take place earlier. Artifacts (plans, documents, models, and code) will have to be modified, reviewed, and approved repeatedly at each revision. Tactical changes or scope changes will force some continuous replanning. Thus, team structure will have to be modified slightly at each iteration” (Kruchten, 2000).
In fact, although agile processes tend to deliver software more quickly than other approaches, the overall timeline of the project is often increased. The time spent is simply more productive and much more in line with customer needs.
Another implication is to keep iterative cycles short and focused. The goal of each iteration is to implement the most necessary functionality as quickly as possible. Extreme Programming recommends iterations of two weeks or less, although the team I work within implements four to six releases per year. The sooner the team can produce working code and get customer feedback, the more successful the approach will be.
This summary was quite brief. If you need a refresher or have never heard of The Blur Box model, visit www.theblurbox.com or look over the proceedings from the 2001 PMI Symposium.
Strategies for Implementing an Adaptive Process
Read Up—Get the Real Story
Agile methodologies are a recent trend, and as such there are no established, time-honored references. Several recent articles in highly prominent publications have sorely misrepresented agile concepts, and you owe it to yourself to go to the source. There are great reads that are absolutely worth your time and which will logically and honestly give you the agile story.
The first is the aforementioned A Practical Guide to eXtreme Programming by Astels, Miller, and Novak. The key features of this book are found right on the cover: making XP principles work in the real world, best practices through the entire life cycle, participant roles, and specific solutions to transitioning problems. If you are even considering exploiting the benefits of XP, this book is a must read.
Another critical source is Agile Modeling: Effective Practices for Extreme Programming and the Unified Process by Scott Ambler. The title is misleading because the book delivers more than promised. This book is a how-to for getting benefit out of agile processes and a survival guide for avoiding common pitfalls. Ambler has a logical, experienced voice that puts the problems and solutions on the table.
Don't forget the web: www.agilemodeling.com, http://martin-fowler.com/articles.html, and http://www.agilealliance.com/arti-cles/index are active and fertile resources.
Evaluate Your Development Team
One of the primary values in the Agile Manifesto is people over processes. To successfully implement an adaptive approach, you must have an internally motivated team that is comfortable taking the responsibility for implementation. Though politics are unavoidable in business, it is best to foster a politic-free development environment. A major reason why agile approaches are successful is that it fosters a collaborative, supportive, and focused atmosphere.
If you have such a team, it is largely self-policing. Individuals may be inexperienced, lazy, stressed, unaware, or otherwise disposed to cut corners or write bad code. But in an agile team, the code is owned collectively and reviewed often by other team members via paired programming, code review, frequent testing, etc. Because the burden is placed on the team, the team often takes more pride in their collective work. Not all team members need to be experienced developers, as long as there is a core of experience within the team.
This leads to another critical element: an intelligent, visionary Chief Architect and/or a core team of experienced developers who have internalized good design principles over the years. Agile approaches require this type of experience because trimming a process to its essentials is only successful when the team can recognize the wheat instead of the chaff. Agile approaches do not circumvent traditional processes such as planning or modeling, they advocate doing only what is necessary for success.
Deal With Your Corporate Culture
If you can't change your corporate culture, at least be prepared to deal with it. If you are successfully following an agile methodology, you will notice the following changes:
• Instead of large directories full of Visio diagrams, user manuals, and other design documents, you will notice whiteboards full of sketches and piles of note cards on the table.
• Managers will not spend time creating Gantt charts and other staffing plans.
• Customers or their representatives will be sitting in design sessions with your programmers.
• The development team will tell the managers how functionality will be implemented instead of vice-versa.
• When asked what the six-month plan is, the best answer will be “I have no idea.”
If any of these changes will rock your corporate boat, you will need to address them head on. An agile process works precisely because it doesn't do the same thing we are doing now.
To be agile, managers, developers, and customers will be working together in different ways. There is no substitute for experience, but it may calm the waters if affected stakeholders can be given a clearer idea of what to expect. I suggest using a metaphor or a model to describe how the changes will affect current processes. Our team uses The Blur Box. At Chrysler they used an assembly line metaphor. Find a common metaphor or model to present to everyone, to get a common dialog going.
Start Small
My wife is a quilter. She makes dazzling bedspreads with intricate designs and colors. But she didn't start with the knowledge for how to do so. Her first quilt, she decided, would be a hand-stitched queen-sized bedspread. That first quilt took years to complete and was quite frustrating. Had she started with a potholder, she would have learned many basic techniques that would have made the bedspread much easier to handle. She could have thrown the potholder away without remorse.
You can do the same with implementing an agile process. Select a test project that your team has some familiarity with, but has uncertain requirements. Implement an agile approach such as eXtreme Programming, or Scott Ambler's Agile Modeling. Be prepared for false starts and stress as the new way of working becomes clear.
It only makes sense to start this way if you are dedicated to pursuing agility. Even a test project is quite resource intensive. Training in the new tools and development processes takes time and money. Also, your team must be dedicated to this process only; going agile is a mentally exhausting effort.
Be Strong in the Face of Success
Ironically, at the beginning an agile team's greatest danger may be success. At first, it seems like nothing will be delivered and everyone gets nervous. “…Suddenly people realize the whole thing is going to be pulled off and begin to fight for their functionality making the first release. The manager must remain steadfast to promised features, or risk jeopardizing the entire project” (Kruchten, 2000). Another danger of success is the team getting overconfident in their newfound abilities. An agile approach is meant to foster flexibility and good code; it is not a way to cram more stuff into a shorter time period.
A Closer Look: The Impact of Iteration on UML
The previous section discussed some general strategies for tackling an adaptive process. What if you are currently using a standard, such as the Unified Modeling Language (UML), and you'd like to also use an agile approach?
Fowler (2001) has recently clarified some of the business reasons for pursuing an adaptive process (what he refers to as evolutionary design). There are two basic types of design: planned and evolutionary. The most common implementation of evolutionary design is code-and-fix, a nightmare that leads to unmaintainable, patchy code. Planned design was to be the panacea to this rampant vigilante coding. In planned design the software is designed up front in abstract terms, and coding is almost an afterthought. The flagship of planned design is UML. The system is mocked up in use cases, association diagrams, graphical user interface mockups, and other conceptual documents before coding begins. This approach does eliminate code-and-fix and brings a unified design to the table, but there are two major problems: the impossibility of thinking through all issues and changing requirements.
In the first case, programmers stumble across issues that question the design, so they code around the problem to meet the design (Fowler, 2001, p. 44). This is ultimately unhealthy for the software. Barrett (1998) writes that it is impossible to design a complex system in one try. If each programmer found just one issue that questioned the design, and each programmer coded around the design, where is the design?
The second case, changing requirements, is a business reality. “The only systems for which requirements will not change are those that have no users, no customers, and no stakeholders” (Davis, 2000, p. 1). Fowler (2001, p. 44) indicates that many planned design advocates attempt to build flexibility up front into the design, but there is no way to predict what the requirement changes will be. At best, the extra time spent on predictive flexibility is wasted; at worst it puts blinders on future design when requirements change differently than expected.
The solution to incorporating planned design approaches such as UML with iterative methodologies such as Extreme Programming (XP) is to use UML diagrams selectively and know why you are using them. For example, developers should not develop use cases for every design session. “We only create use cases the first time we are approaching a problem or the first time we work with a specific customer. After we understand the problem or customer's business practices, the use cases become cumbersome” (Baker, 2001). Fowler (2001, p. 46) advocates this approach: “This selectivity is the key to using the UML well. Don't draw every class, only the important ones. For each class, don't show every attribute and operation, only the important ones…You get the picture.”
Grady Booch, one of the authors of the UML, has recently weighed in on these issues. He admits that UML, in becoming a standard, is being used and applied in inefficient ways to the wrong projects: “As for modeling, UML has definitely entered the mainstream, and is being applied in places far beyond its initial goals” (Booch, 2001). UML is certainly a great tool, but it does not work as well for projects with high flexibility and iterative requirements. “…heavyweight processes…dictate the rhythms of individual developers as well as the team as a whole. The problem with this is it typically stifles creativity and leads to horribly over-engineered solutions” (Booch, 2001). The goal is to find the sweet spot of just enough overhead. The key is to do only what is necessary for the success of the specific project.
Conclusion
Agile approaches have so much to offer: they prevent late discovery of critical problems, they promote collaboration among stakeholders, they allow developers to focus on the most critical features, and they enable flexibility and responsiveness. To exploit these benefits, managers need to have a good idea of what they are getting into. Only with a clear understanding can managers plan effective transitions and avoid the trap of following unnecessary predictive practices.
This goal of this article is to get managers thinking about how to successfully approach an adaptive process. One of the core values of XP is Courage; it takes real courage to announce that a much loved and time-honored process simply isn't doing the job. Managers will need courage to get through an agile implementation. I hope this article gives you the courage to tackle an agile approach, or at least has given you insight into some of the issues involved. Read, stay informed, and evaluate how and when an agile approach might help your company reach its goals.
