Diving off the waterfall into agile

Abstract

How can an organization effectively transform from a traditional waterfall to an agile delivery approach? This presentation covers how we adapted our project delivery processes and team structure to provide value more efficiently to our internal customers. We will discuss our challenges and how we addressed them.

Introduction

Diving into agile from a traditional waterfall approach is a change that many organizations find daunting. With backlogs, epics, user stories, and scrum masters, the terminology alone can intimidate a traditional project team. Changing mindsets to adopt an agile approach is as much a cultural shift as it is a change in technology or project management. Becoming agile is a journey that highlights inefficiencies and takes people out of their comfort zones. It challenges people to think in new ways and to change working habits.

The process can be rocky with challenges such as team adoption, revised project roles, shifting priorities, changes in scope, resource capacity, status reporting, leadership demands, and customer feedback. However, traversing the waters of agile can result in greater efficiencies, innovation, process improvements, team collaboration, and improved customer delivery. The key is to do what works best for your organization and your customer, incorporating agile principles in a phased approach and where they make sense. Dive in where you want and land in the approach that fits your project.

Waterfall and Agile

Before we discuss our journey, a few points around the myths and facts of waterfall and agile will provide context for our approach.

Common Perceptions

• Waterfall is completely outdated.
• Agile is new.
• Agile is undisciplined.
• Agile projects do not produce documentation.
• Agile is easy.

The Reality

Waterfall is somewhat outdated. The market is certainly demanding faster, more flexible development approaches than waterfall has traditionally been able to support. However, not every organization is prepared for the empirical processes that agile promotes. The emergence of “hybrid” approaches, typically encompassing iterative requirements, design, development and test cycles and application of agile techniques within a traditional, predictive project management approach attempt to serve this need with varying results.

Agile is not new. “Agile” methodologies evolved in the mid-1990s, and the term “agile” came to encompass these methods with the publication of the Manifesto for Agile Software Development (Beck, 2001). While the methods and terms vary, they all have a focus on adaptability, collaboration, frequent delivery of business value, self-organizing teams, and acceptance of change.

Agile is not undisciplined. Because agile provides greater transparency and visibility into progress, it demands greater discipline than traditional approaches. Status is tracked and reported daily – not weekly. Planning is done continuously. Requirements evolve through collaboration in real time. Teams are required to self-manage, which requires a very high amount of discipline and is a source of failure for many teams new to agile.

The Manifesto for Agile Software Development values “working software over comprehensive documentation,” which is sometimes interpreted as no documentation is needed. Agile projects produce the documentation that is needed – and only what is needed – which may vary considerably from project to project. For example, a project in a highly regulated industry may have significant documentation requirements whereas a gaming app for a mobile device may have minimal documentation. Documentation may also be produced at a different time as compared to traditional projects. Rather than investing in documentation up front and maintaining it throughout the development cycle, agile projects may focus on delivering the working software first and completing documentation once the software is stable and accepted by the project team and/or stakeholders.

Some believe agile is easy to adopt because of the perceptions discussed above. Nothing could be further from the truth. Agile done properly can be incredibly difficult for organizations to adopt, especially large organizations with a long history of waterfall development. It requires changes in management expectations, project funding, contracting processes, organizational structure, and cultural norms.

Our Organization

Our organization is probably similar to other organizations. We are in consulting, but we are focused on the creation of methods and tools for our consulting practice. This includes the support of our published methods and tools in addition to the adoption, integration and innovation of our solutions. Therefore, like many organizations, we work on projects to innovate our methods and tools, but also have changes, suggestions and ideas received through feedback from project teams that need to be prioritized and worked.

Our Journey Starts

The journey from waterfall to agile has been a lesson in project management. Each year we improve in the areas of strategic planning, work planning, team organization, work management, and status reporting. This has enabled us to operate more efficiently and to provide timely solutions to our practice.

We started our journey like many companies – planning, designing, developing, and delivering in a traditional waterfall approach. We focused on developing detailed work plans, tracking progress, and analyzing schedule performance and deliverable completion rate. With this experience in place, we defined our own standards for project management to try to achieve a consistent approach for managing projects, while enabling flexibility for variations in project size and type. The primary usage models included: Basic, Effort Managed, and Actuals Managed, as shown in Exhibit 1.

Exhibit 1– Work plan usage models

Our Case for Change

We continued to challenge ourselves to improve our approaches to deliver updates more efficiently to our practice. We needed to think of ways to evolve to address three specific challenges:

Ability to prioritize and reprioritize our work

One of the main areas we had to address was our ability to prioritize and re-prioritize our work based on the evolving circumstances of our practice. In our traditional waterfall approach, we scoped and planned our projects for major releases, developed charters, estimated the work, planned the project and established the teams. After we planned the work, we performed the detailed design of our solutions; however, we encountered requests for new, higher priority initiatives. Our scope control processes were invoked, but this re-prioritization caused the previous detailed designs to be put on hold or canceled.

Team structure

We defined our teams defined and assigned people at the start of the year based on a planned set of priorities and projects; however, the work re-prioritization required us to adjust our teams. The effort spent defining and on-boarding the teams for large, yearly releases did not align when the priorities were changing.

Provide value more efficiently

Like many organizations, we are evaluated on the impact of our projects on the organization and the efficiency of the delivery of the impact or value. While our processes worked well for large, yearly releases, we were still challenged to get value and impactful solutions out much more efficiently. We needed a way to release content more iteratively to the practice.

Our Journey Continues

A more agile-like approach seemed to be the solution. We shifted gears and moved from detailed work plans to high-level milestones. We created a prioritized list and roadmap with specific points throughout the year where updates could be realized. This forced us to change operations in five key areas as shown in Exhibit 2.

Exhibit 2 – Differences between our waterfall and agile approaches

Sounds easy, right? On paper, it works well, but in reality, it is a big change for the organization. Leadership has to support an agile approach and prioritize the efforts, managers have to change the way they manage, and resources have to change the way they work. It is not an overnight fix. One key to making an impact with agile is to implement it in stages: start by introducing agile concepts and translate current processes to the agile way of thinking. Give people time to learn new tools and terminology. At first the organization may become less productive; however, it will likely improve as everyone starts realizing the potential benefits of the approach.

Our journey is not over. Our organization continues to adopt agile concepts in internal processes and to apply sound project management practices to a non-traditional project management approach. With the combination of both waterfall and agile approaches to managing projects, our practice can in turn choose what works best for their clients. It is about delivering value and responding to ever-changing business circumstances.

Lessons Learned through Retrospectives

The chart below identifies specific lessons learned in the waterfall-agile journey. We hold regular sprint retrospective meetings and continue to drive these lessons to keep the team operating efficiently..

Exhibit 4 – Lessons learned through retrospectives

Wrap-Up

Software development and our approach to project management is changing. As you embark on your own journey towards increasing agility, consider the following:

Expect Common Challenges

Organizational Commitment and Collaboration

Agile projects require the ongoing collaboration and commitment of a wide array of stakeholders, including business owners, developers, and security specialists. Challenges in achieving and maintaining such commitment and collaboration may include:

• Agile may expose organizational opportunities for improvement;
• Mixing agile with “Command-and-Control” management cultures can be challenging;
• Teams may have difficulty collaborating closely and committing to frequent input;
• Teams may have difficulty transitioning to self-directed work; and
• Changing culture and process can be challenging (and may be costly).

Preparation

When an organization following waterfall software development migrates to agile, new tools and technical environments may be required to provide additional guidance for the approach, as well as updates to guidance and procurement strategies. Some challenges in preparing for agile include:

• Timely adoption of new tools can be difficult;
• Technical environments can be difficult to establish and maintain; and
• Procurement practices may not support agile projects.

Execution

Agile projects develop software iteratively, incorporating requirements and product development within an iteration – called a sprint. Such requirements may include compliance with legal and policy requirements. Challenges in executing steps related to iterative development and compliance reviews are as follows:

• Teams may have difficulty managing iterative requirements; and
• Compliance and regulatory reviews can be difficult to execute within an iteration time frame.

Evaluation

Agile advocates evaluation of working software over the documentation and milestone reporting typical in traditional project management. Challenges in evaluating projects related to the lack of alignment between agile and traditional evaluation practices are:

• Some reporting practices do not align with agile; and
• Traditional artifact reviews and status tracking do not align with agile. Hybrid approaches may be needed.

...And Preparation is one of the Keys to Achieving Goals

With proper planning and management of expectations, you can achieve the desired results. Some ways to facilitate your journey might include:

• Train stakeholders in your agile approach, and train your team in your agile methods;
• Create an environment, whether physical or virtual, conducive to collaboration;
• Identify measurable, value based outcomes - not outputs - of what you want to achieve using agile;
• Take steps to align your contracts or project funding processes to accommodate your agile approach;
• Include staff with agile experience or supplement with coaches where needed;
• Prepare for potential difficulties, regression, and uncertainty;
• Migrate to agile maturity in-sync with your readiness;
• Determine project value based on customer perception and return on investment; and
• Understand this is a journey and the process will take time, but increasing the agility of your organization can bring many improvements.\