In this paper, the authors present their hypothesis that in the future, projects will use a combination of agile practices, with a traditional approach, usually associated with waterfall projects. This should become the dominant successful methodology, even despite the pressure for quick deliveries and the unstoppable growth of agile philosophy. The article includes specific recommendations on how to blend agile and waterfall approaches in aspects such as scope, schedule, communication, teams, and others.
Keywords: waterfall and agile, hybrid agile, agile with discipline
INTRODUCTION
Involve the customer—the involvement should be formal, in-depth, and continuing.
Dr. Winston Royce, 1970
The above quote comes from the foundational paper on waterfall methodology. As you can see, agile did not invent customer involvement. The issue is that bad waterfall projects have created the wrong common wisdom on what waterfall really “was.” Therefore, we tend to compare agile projects to bad waterfall approaches and continue with this misconception.
The financial services industry provides a very good example of the success of agile as the dominating philosophy over the waterfall approach, and it is the agile “brand” value proposition. The banks have decided to change the image they project to societies around the world (remember the—yet to end—2008 crisis) by not only engaging in a total new relationship with their customer base (they will be your financial advisor, money management educator, and your financial “coach”) but also dramatically changing the way they have been delivering solutions for the last 30 years. The new value proposition by financial services institutions comes bundled with a universal program of embracing agile 100%.
Therefore, 21st century banking innovation comes in two dimensions: a portfolio of new services and customer intimacy to enable the banks to compete with new financial players Apple, Amazon, Google, and so on, in all money-related initiatives, on one side, and a total transformation of IT processes (which basically count for almost 100% of banking processes) into agile programs (all major banks have one) where the word “waterfall” is nearly a blasphemy. Therefore, no questions asked: you are “agile,” therefore you are cool, young, flexible, adaptive, disruptive; you are “waterfall,” therefore you are old, failed, dictatorial, bureaucratic, out of time. In a world where you are supposed to do everything by tapping on a smartphone, agile is obviously the only way to go. However, as the Greek philosopher Aristotle indicated long ago, the middle ground between extremes, or the Golden Mean, is probably the right approach to successfully implement the challenging projects ahead. In a recent conversation I had with a bank executive, when I estimated a three-month implementation (inception to go-to-market) for a relatively simple project, he responded: “No way, we need this live and out there in a few weeks.”
With almost no time to think, or to write, there is no room anymore for those 80s or 90s projects with functional design, technical design, architecture design paper binders, but just for a good—if possible—product backlog, and while many companies are mixing agile and waterfall approaches, they do it almost—at least in the financial services sector—with the agile “brand” and approach as the dominant one, and waterfall approaches are often hidden and even despised sometimes. We try in this paper to develop the key aspects of blending these approaches, even if you want to keep the waterfall “ingredients” undercover and embrace (as you should do) the unstoppable agile world. We have structured this paper in specific recommendations that address all aspects of a project, to explain our hypothesis that a hybrid agile/waterfall model will be the lasting approach to most projects ahead. The idea is already spread throughout the industries, and a good example is the disciplined agile delivery framework for software development developed by Scott Ambler and Mark Lines in 2012.
STAKEHOLDER MANAGEMENT: EVEN WITH A NEW TITLE, AGILE PROJECT STAKEHOLDERS SHOULD USE THE OLD WATERFALL DISCIPLINE FOR THE INITIAL PROJECT DEFINITIONS
The blending of waterfall and agile should occur at the beginning of the project, when, for example, in the Scrum methodologies, a product backlog must be prepared. Even though this product backlog may not be fixed in stone, our experience shows that most successful projects used the frequent delivery approach of agile (two weeks, four-week sprints) but with a robust preliminary definition of what would be the final product, or “customer experience.” In a bank that recently implemented a successful financial management project in just four months, a key success factor is that the so-called customer journey, and therefore the product backlog, was well defined, and documented up front; in a waterfall world, we would have said that requirements and design phases had been well completed, with a high level of quality. The project enjoyed the agile approach by having product backlog deliveries every three weeks, and changes to that initial backlog were implemented easily, but just because the initial documents where thorough and as we sustain, we applied “waterfall attitude” to the initial stages.
A clear definition of roles and a disciplined way to develop the initial project deliverables is indispensable for a successful project. In agile projects, the product owner must develop documents and designs for the product backlog with a waterfall approach. The mere use of user stories without a robust document that envisions what the minimum viable product (MVP) will be, will make the project prone to scope creep and unnecessary delays. Since a product owner acts as the business representative and the voice of the customer, there is no possible approach to unify and integrate all the key stakeholders’ specific expectations through a structured waterfall-like initial design. For the rest of the project, the attitude must switch to a more informal, interactive relationship between the product owner and the agile team. And a key agile principle that has to be respected (and it is sometimes disregarded) is that flexibility goes both ways. A product owner must understand that if any item cannot be delivered in a specific sprint, it will go for the next. Otherwise, the agile project turns into a series of time-boxed waterfalls that make life more difficult in an agile project than in a traditional waterfall. I have heard the sentence: “You failed to deliver -x- in this sprint” quite a few times.
Putting human interaction above contracts, as agile philosophy sustains, means a more flexible approach on the part of project sponsors and business users. “I am flexible to understand that you don't know everything you want, up front, then you should also be flexible in accepting that not all deliveries will be made exactly as planned.” As we already know from those old waterfall times, the important achievement is to deliver the project on time, not all intermediate milestones on time, unless really critical. Bad agile implementations may turn an agile eight-sprint project, into an impossible program of eight inflexible waterfall projects.
In Scrum projects, the project manager role can be assumed by the ScrumMaster, but experience shows that a project manager, in addition to the ScrumMaster who liases the Scrum team with the product owner, can bring a lot of value, especially when the people have not experience in agile projects. The role of a project manager is also paramount in defining the initial vision for the project, and since the evolution of the project will be based more on interaction than in documents or processes, the soft skills of the agile project manager are much more important than they were in the waterfall approach, where documents, contracts, and published processes were of higher importance than the (sometimes excessive) day-to-day interaction of agile. However, at the beginning of the agile project, when sponsor and product owner envision the high-level objectives, a waterfall mindset for the definition of the minimum viable product and the product backlog is advisable.
For this, a good method is to open a so-called “Sprint 0” (be sure you keep agile jargon at all times) of two weeks to two months, where creativity, innovation, lateral thinking, and all team brainstorming activities applicable will take place in order to build a minimum viable product that—sometimes silently—the project manager (“agile with discipline”) integrates into a backlog table of requirements. Yes, in paper.
HYBRID PROJECT REQUIREMENTS PROCESS: MERGE USER STORIES INTO A ROBUST AND STRUCTURED PRODUCT REQUIREMENTS TRACEABILITY MATRIX
There is a number of unjust criticisms against the waterfall or “traditional” approach; in one of the best books about agile planning, Mike Cohn (2006, pp. 15–17) states that:
A critical problem with traditional approaches to planning is that they focus on the completion of activities rather than on delivery of features.... another reason why traditional planning fails to lead consistently to high value products is because the work described by the plan is not prioritized by its value to the users…
This would happen in badly managed waterfall projects, but nothing and no one prevents you to develop a waterfall plan with a properly deliverables-oriented work breakdown structure (WBS) and schedule. When addressing the first activity in an agile project, putting a strong effort in defining an itemized product backlog is a great jumpstart:
- ID
- Product feature/User story
- Business purpose
- Priority
- Sprint no.
- Activity/Deliverable no.
- Supporting document ID, and so on
The difference with waterfall is that a change control procedure may not exist; but then, no hard deadlines and low budget constraints can exist either. Again, you may change as much as you like in this backlog requirements table, but don't impose hard milestones for the whole scope, since, as we said before “flexibility goes both ways.”
HYBRID PROJECT SCHEDULE DEVELOPMENT: THE AGILE PROJECT MANAGER MUST KNOW AND COMBINE CRITICAL PATH TECHNIQUES, CRITICAL CHAIN TECHNIQUES, AND TIME-BOXING APPROACHES
Customers have not patience. In today's competitive world, time to market is a key factor for success. So don't be naïve, they want the speed of agile, combined with the predictability of a thorough and traditional project schedule, (but make sure you don't call it waterfall) that eases their anxiety in launching new products and services to an increasingly demanding customer base. The competent project manager should collaborate with the product owner in helping prioritize (for example, with the 1980s KANO technique) the project deliverables with the ScrumMaster in building sprints and the forecast of releases, but also needs to spend some significant time building a long term (the length of a complete project life cycle) project schedule that anticipates potential slippage.
Due to the nature of agile projects, project buffers must be applied here almost with no exception, otherwise the whole project may fail. If, for example, the business intent is to launch a product or service for customers very quickly, the agile project manager must build a very robust schedule to deliver the minimum viable product (MVP) to market in that time frame, and be very strict with new “must haves” that may arise through the sprints making the business and sponsor aware of the potential slippage of the MVP launch. This may make many stakeholders in the project believe that “This is not agile,” and here is when the word “hybrid” and “blended” approaches appear, in the discussions between the agile teams and the product owner and project sponsor. What-if scenarios in managing schedules will be essential. This type of skill is very well developed when working in waterfall projects. When a “quick win” is necessary (and currently all business ideas need this) you may be waterfall—undercover—for the first launch and pure agile for the future evolution of your solution afterwards, because you already have something in the market and you gained the credibility to apply agile 100%.
APPLY VISUAL MANAGEMENT TO PROJECT TEAMS: “MILLENNIALS ARE DIFFERENT”
One of the titles I thought as possible for this paper was “Too old to agile, too young to die” as in an old Jethro Tull song; reality is that now in my fifties, I face some difficulties in understanding the new generations, and while apologizing for the clichés I may fall into, I would like to include some advice here on how to organize teams for agile projects and combine old waterfall methods. That reading habits are diminishing is a proven statement, and what mainly distinguished the 80s and 90s projects I personally worked from the ones I overlook now is the amount of binders with hundreds of pages that were read, revised, amended, and became the “scope definition” at that time. Young people prefer interaction and “test-driven development” to documents, in general, and in particular, the Agile Manifesto (2001) made it clear:
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
This is undeniably true. We fail to express on paper or with images all that we intend to achieve in a project. But we find a paradox here: the new generations express themselves more through social-networking apps such as Snapchat than through emails, let alone long documents, but also they use calm conversation or fruitful constructive dialogue and the continuous conversation that agile requires in the absence of long documents less. An approach that was already used in waterfall projects, and that combined with the product backlog covered before helps this interaction is the use of prototypes. Young agile teams love to have conversation over prototypes, mockups of what the product owner wants to receive. Young agile project teams are more image-oriented than text-oriented. They are more visual than conversational. Therefore, the project manager has to bring good drawing, sketching, and visual thinking skills into the agile project. We can't really substitute bulky waterfall documents by conversations and interactions only, and use of visual transfer of knowledge (i.e., requirements) is highly advisable.
DO USE EVM IN AGILE; PROCESS AND TOOLS ARE ALREADY THERE
Are agile projects inventing earned value management (EVM)? Many of the so-called innovations brought by agile methodologies could have been implemented for waterfall—and in many were indeed—including team colocation, deliverables orientation, progressive elaboration of scope, and so forth. A common mistake we already discussed in this paper is that the agile literature is comparing agile projects to bad waterfall projects; and for the new generations that probably did not see any waterfall project at all, they have been delivered this binary message: “Agile Good, Waterfall Bad,” and that is oversimplifying reality. I remember projects in the early 90s where we did “extreme programming” with no documented specifications whatsoever, but we did not decorate this with any name. I also remember the end of 90s with Euro implementations and the 2000 panic syndrome where many projects were delivered in a “very agile” way. Now I find for example, that we use some tools that use EVM techniques in agile projects to display progress and forecast the evolution of the project. Since these tools are being implemented by agile software tools vendors, it is very probable that I will hear somebody in the near future saying “You see how cool agile is, agile invented EVM, and it is a great technique.” The agile projects rely on burndown charts (but this may change) and velocity assessments. But using user story points, on a sprint and project basis, and its combination with EVM, we will get the best of both worlds: for example, schedule performance indicator (SPI)=user story points completed/user story points scheduled; cost performance indicator (CPI)=user story points completed/equivalent point=hours spent. The agile-friendly tools in the market are already implementing these formulas aware that they are as efficient if not more than burndown and velocity charts, because they allow forecasting not only for the sprint but for the whole project.
CLASSIC PLANNING PROCESS IS WELL PRESERVED IN THE AGILE ENVIRONMENTS. IT IS JUST HAPPENS MORE FREQUENTLY. BY A. RODOV
Consider waterfall scheduling process as a complete set of steps to produce all necessary documentations, communication, commitments, and stakeholder anticipations. Well, sounds like a lot of steps, but this is what we've figured is best for complex projects to do and the Project Management Institute taught us well to follow the process and established it as a good practice. I have a technical background and thanks to my teachers who taught me how to translate processes into mathematical models and see if there are benefits to apply rules to scale out the model and yet preserve the model outputs based on the model inputs. In other words, fundamental planning process is well preserved in agile environments. It just happens more frequently. Look at Exhibit 1. It illustrates clearly that incremental planning will let you be more flexible, and spend less work marching toward the final project objective.
Recommendation: Increments can be as small as weeks or days as well as large as months or quarters. Use the approach above to understand how much effort is spent on managing incremental changes versus the benefits you get from doing that. For that, you may need to have some statistical data to come up with what would make sense from the institutional culture, historical process adoption, and so forth.
Exhibit 1: Waterfall versus agile incremental planning.
Do not create too small and too large increments in order to get the benefits of incremental planning as well as keep existing organizational process aligned.
INFORMAL COMMUNICATIONS. THE CONVERSATION.
Collaboration is very much needed regardless of whether you prefer agile or waterfall, or both. Among the tools helping with communication, there are the following formal and informal methods of communication: new tools and methods of communication create additional discipline of managing these channels and understanding that the outcome of this communication good for project. The more informal communication your teams get to experience the more efficient this communication becomes. Basically if you are at a meeting with 20 people, likely only two or three people efficiently communicate. The rest of the attendees are probably not as efficiently engaged or are checking their Facebook page. With informal communications, the conversation is almost always one on one. You have a chance to listen and talk and exchange ideas quickly and fully. In this paper, I would like to embrace informal communications, as it can and should be adopted by any project team. See the example of formal communications versus informal in Exhibit 2.
Exhibit 2: Informal communication.
Lines from the center dot represent formal communications, and lines between other dots represent informal communications. The more connections between individual dots the better for project communications. Online chat, daily standups, Facebook, Yammer, and SharePoint collaborations will do the trick. A good old phone call may also be worth a thousand words in a chat window. However, don't forget about mobile communications. This is becoming really big. The number of things people do on the phone is growing, and project management communication is one of those many. See Exhibit 3.
Exhibit 3: Communication/time spent on mobile (comScore, Inc., 2016).
Recommendation: Embrace informal communication and establish a comfortable communication ecosystem within your teams. Collect the communication and make it accessible and sharable. Use tools that work with mobile platforms.
ABOUT THE AUTHORS
Alexander Rodov, PMP, Microsoft MVP, MCTS, CSM, was awarded the distinguished Microsoft MVP award in 2014 and 2015. Over the past 12 years, he has specialized in assisting CIOs, CTOs, and PMO directors with deployment of Microsoft's industry-leading enterprise project management solution, including Project Server, Project Portfolio Server and all closely related Microsoft Software. Alex holds a master's of science degree in Mechanical Engineering and Computer Science. He has been a Project Management Professional (PMP)® since 2008 and is a Microsoft Certified Technology Specialist as well as a Certified ScrumMaster.
Jordi Teixidó, PMP, is a project management consultant and speaker with over 25 years of experience managing international projects. A former Accenture senior manager, he is the founder of KION, a training and consulting company and PMI Reglistered Education Provider he is currently working as Chief Operating Officer for Strands, a world leader in Digital Transformation in the financial services arena, and one of the top “FinTechs” worldwide. He also teaches graduate and postgraduate courses in project management at the Polytechnique University of Catalonia (UPC) and gives lectures on management around the world. His current motto is “Too old to Agile, too young to die.”
CONNECT WITH ME!
Linked In https://www.linkedin.com/in/rodau
Linked In https://es.linkedin.com/in/jorditeixido |
@Twitter: #alexrodovpro
@Twitter: #JordiTeixidoPMP
Ambler, S. W., Lines, M. (2012). Disciplined agile delivery: A practitioner's guide to agile software delivery in the enterprise. Boston, MA: IBM Press/Pearson.
Cohn, M., (2006). Agile estimating and planning. Upper Saddle River, NJ: Pearson.
comScore. (2016). Media metrix mulit-platform & mobile metrix. Retrieved from http://www.comscore.com/Products/Audience-Analytics/Media-Metrix-Multi-Platform
Royce, W. W. (1970). Managing the development of large software systems. Retrieved from http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf
The Agile Alliance. (2001). The agile manifesto. Retrieved from http://www.agilemanifesto.org/iso/en/principles.html
