Abstract
Ten years after the signing of the Agile Manifesto, agile methods have gained significant mindshare, attention of the industry, and public interest, across the software product development community. The promise of agile is great and the benefits are well demonstrated, and many people are open to considering an alternative approach. The challenge is that most organizations are, in practice, not agile at all. They may have teams using agile methods, but are in fact, hybrid agile and traditional organizations. Very often, agile at the team level is not enough to achieve true end-to-end business agility. Unless the business gets the benefit of team level agility, the investment in adopting agile is meaningless. This paper explores what it takes to fully adopt agile in an end-to-end fashion, allowing the enterprise to inspect and adapt and change course while minimally disrupting the product development process.
Introduction
Agile methods are gaining mindshare and popularity across the project management community, but are primarily relegated to the field of software development and product delivery. The case can be made that product delivery isn’t fundamentally project work and requires a much different planning paradigm, but the reality is that projects are how much of product delivery is handled in traditionally managed organizations. In either case, understanding where agile methods fit, and how these concepts can be applied and scaled to the program and portfolio is of primary concern among many practitioners.
Over the past 10 years, since the signing of the Agile Manifesto, many of these software product organizations have explored introducing agile methods, and agile project management techniques, with the conviction that agile will indeed scale to the enterprise. Although the core concepts and principles behind agile will scale to the enterprise, many of the practices are better suited for small teams of 6 to 8 people working on smaller initiatives in relative isolation from the rest of the organization. Some of these organizations have come to the conclusion that agile does not work at scale.
To begin to address the enterprise aspects of agile at scale, organizations need to draw from a broader palette of tools and techniques that support not only agility at the team level, but also enable business agility across the broader organization. It is also important to note that agility at scale requires a certain degree of pragmatism, and a realization, especially during the early stages of introduction, that the organization is going to need to blend approaches to be successful. Understanding the transformation roadmap, and where your company is along the way, is of critical concern.
The end state of any enterprise agile transformation program, by necessity, will impact almost every aspect of the business. Ultimately, enterprise agile means making smaller investments at every level of the organization. Portfolio investments are smaller, program deliverables are more discrete, and project requirements are fine-grained and crisply defined. The goal is to create a continuous flow of value across the delivery organization that enables fast feedback, the ability to quickly change direction, and to progressively reduce risk and validate assumptions as the product is built.
With a proper understanding of business agility, the foundational concepts underlying agile methods, and a broader palette of tools and techniques at their disposal, the PMO is well positioned to drive value creation and govern the delivery process across the organization.
Goals of Project, Program, and Portfolio Management
To understand how to best evolve the practice of project, program, and portfolio management, it is helpful to first understand the goals of the practices and how they relate to each other. In order to achieve business agility, a systems view of the entire body of work is required.
Project Management
Project management is fundamentally concerned with managing the duration of the project, the overall cost of the project, and the deliverables that the project stakeholders expect to see from their investment in the initiative. Balancing these triple constraints, managing the day-to-day project activities, continuously assessing risk, and validating assumptions to ensure stakeholder expectations are effectively managed, together defines much of the roles project managers play in the typical product organization. Aside from the hard skills associated with project management, it is generally recognized that soft skills play an equally important role in successful project delivery. Successful project managers are leaders, team builders, and great facilitators and often become the glue that binds a project together.
Program Management
Program management fundamentally addresses a similar set of concerns as project management, but at a different level of scale. Although the project manager may own a discrete set of deliverables, with a well-defined beginning and end, the program manager is responsible for a collection of projects that must be integrated over time to achieve a larger business goal. Quite often, programs will span multiple releases of a single product and require a longer-term, more strategic perspective on the life of the project. Each release has to be self-contained, but also set up the organization for any subsequent releases that are in scope at the program level. Program managers are still dealing with time, cost, scope, and risk associated with project deliverables and leadership is an even greater concern.
Portfolio Management
Portfolio management takes an even broader view of the project/program ecosystem. Portfolio management considers the business goals of the larger organization and approves funding for the various initiatives in which the organization may choose to invest. Where a project or a program has approved resources and funding that must be appropriately managed, the portfolio manager is facilitating the overall understanding of organizational capacity against demand, how to assign people and resources, and how projects will be interleaved to optimize business outcomes given a limited set of people and resources, and usually more demand for those resources than the capacity to deliver.
Project, Program, and Portfolio Governance
Traditional versus Agile
Traditional approaches to project, program, and portfolio governance tend to focus on the idea that with sufficient up-front planning, disciplined process control, and appropriate change management; project outcomes can be known in advance and that the organization must simply work the plan to completion and business outcomes are achieved. Even from a traditional governance point of view, this thinking may be considered simply poor management, but quite often this is the model driving many software product organizations today.
Agile approaches to project, program, and portfolio governance tend to focus on the idea that project outcomes cannot be sufficiently known in advance, and the planning and process overhead associated with traditional approaches is waste when uncertainty and change are the normal state of affairs. Furthermore, when markets, technology, and customers frequently change the competitive landscape, long-term planning discourages change, and limits the ability to respond to change and adapt the organization for survival.
Failure Modes of Agile Methods
Traditional approaches to governance tend to fail in conditions where fast change is normal and in conditions where the business needs the ability to inspect and adapt on a continuous basis. Predictive planning can work, but is often expensive and lags behind what is actually happening on the ground with the product delivery team. Traditional governance will work to slow down and control the process when the business requires the process to be fast and light.
Failure Modes of Agile
Ideally, the organization needs just enough up-front planning to maintain guidance and control, while deferring as much planning as possible to allow the organization to rapidly adjust to changing market conditions. Finding this balance is a key concern when adopting agile at the enterprise scale. Agile approaches often go wrong when practitioners do not acknowledge the legitimate needs of the business to know what they are going to get, when they are going to get it, and when the initiative is going to be done.
Many organizations will struggle to implement Agile governance, when on one hand they ask for speed and flexibility; but on the other, demand precision around investment outcomes. The need to understand what the organization will get for its money and make commitments often competes with the reality that the business needs to frequently change its mind and learn as it goes. This may result from competing agendas among the primary stakeholders and reflective of a business alignment or strategy issue rather than a project, program, or portfolio governance issue.
The Goals of Agile Project Management
Getting Started with Agile
To understand enterprise agility, one must first understand a bit about the history and goals of the prevailing agile methods in practice today. First, agile is not a stand-alone approach; it is more an umbrella term describing a family of approaches. Among these approaches are methods like Scrum, Extreme Programming, DSDM, and Agile Unified Process. In 2001, a group of people who were exploring these approaches got together in Snowbird, Utah, USA, and created a document called the Agile Manifesto. The authors coined the term “agile” to describe what these approaches had in common:
The Agile Manifesto
We are uncovering better ways of developing
software by doing it and helping others to do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.
Beck, Beedle, Bennekum, et al. (2001, www.agilemanifesto.org)
The Agile Manifesto forms the basis for much of the conversations about how to apply a broader set of tools and practices to the problem of enterprise agility.
Empirical Process Control
Ken Schwaber, the co-creator of Scrum, one of the most popular agile approaches, introduced the idea that systems that are highly variable do not benefit from predictive process control. In other words, when the product development process is stable and not subject to change, a project manager can predictively manage outcomes and manage variation through change control and buffering scope, schedule, and cost variables. When the product development is not stable, due to variations in developer competence, changing requirements, uncertainty, or rapid technological advancement; predictive planning is expensive and may not lead to optimal business outcomes. By delivering in small increments, inspecting the outcomes, and adjusting the process as one goes, the organization may end up with a more expensive process, but one that is more likely to deliver the business outcomes the organization is targeting.
Limitations of Agile
Agile solves this problem by reducing as much variation as possible, and then allowing the team to inspect and adapt through the rest. Attributes of agile project teams tend to center around cross-functional participation of all team members, specializing generalists that work together to solve any problem that the team may face in developing the product. Agile teams value co-location and small size as ways of amplifying communication and reducing the overhead associated with keeping everyone informed about what is happening on the product. The general rule of thumb is that an agile team has a single customer representative to guide the development process and is fully empowered to make all decisions regarding the implementation of the product.
The guidance proposed by agile methods that work excellent for small teams, often break in larger more complex organizations, or ones where the organization is not fully aware of how and why agile processes operate. When a team has external dependencies, they are often not in control of their commitments and have difficultly establishing a stable throughput. One common instantiation of this is when the team commits to having an increment of working product, but shares a key person with another team. This dependency has to be managed across teams, which is fine when everything is going according to plan. When one or the other team is delayed, it prevents the other from meeting its goal for that iteration. At scale, dependencies are the norm unless specific action is taken to decouple them.
Even when due care is taken to decouple dependencies, large organizations often have shared service groups, that one or more agile teams will depend on to create a fully functional increment of working software. This sharing of key services, or components within a larger enterprise, is often unavoidable and even desirable. When shared services are involved, it is incumbent on the organization that no one team gets too far ahead of the others before they come together to deliver some larger integrated increment of value. Reducing dependencies between individuals and teams is one of the key challenges of implanting agile at scale; managing those dependencies that remain, while maintaining business agility, is the other.
The Goals of Enterprise Agile Project, Program, and Portfolio Management
Accuracy over Precision
Given that an agile approach is best suited and best applied toward projects where uncertainty is the norm and change is an expected part of the delivery process, the Agile Enterprise seeks to value accuracy over precision. This means that the organization must learn to make commitments at higher levels of abstraction and set delivery windows that reflect the inherent uncertainty in the process. When cost and date considerations are the primary concerns, care must be taken to help the organization converge on the optimal business outcomes possible given the rate of change.
Stability and Predictability
Because agile methods ask the organization to rightfully accept a certain amount of ambiguity in the delivery process, or at least acknowledge the fallacy of predictive up-front planning, each team in an agile organization must be predictable over time. This means that each team should strive to establish and maintain a stable velocity over time, such that the organization can reliably predict the impact of scope changes to the overall program and portfolio ecosystem. If teams are not predictable over time, there is little chance of establishing a stable, predictable delivery capability. This goal requires several things from the larger organization, most notably the creation of teams that stay together over time, estimate together as team, and work from a well-groomed backlog of product requirements.
Rapid Risk Reduction
When approaching a high-risk initiative, every level of the organization should focus on reducing risk and narrowing the cone of uncertainty as fast as they possibly can. Agile teams and agile organizations reduce risk by building working software, getting feedback from customers, and continuously validating assumptions as they go. Failure to do early risk management is a key failure mode for both traditional and agile projects and programs. Software product delivery organizations face constant risk of building the wrong products and basing schedules around unproven or unknown technologies and approaches. Working these high-risk scenarios early is a critical success factor for organizational success.
Fast Feedback
In fast changing environments with high subjective success criteria, getting feedback from the product’s consumers is a critical success factor. At every level of the organization, be it the agile project team, the program management organization, and even the portfolio management organization, the goal is to minimize the amount of time investing money into a project without validating the direction with a real consumer. Teams that deliver incrementally, but do not show their products to customers while they are in development, risk building products that may not suit customer needs.
Easy to Change Direction
As markets change and customer demand ebbs and flows, the organization needs to be able to easily change its mind about what is most important. When change is a normal part of the product delivery life cycle, it is often wasteful to do excessive forward planning. Furthermore, how the organization structures itself, all the way from teams and technology to project investments and corporate structure can either increase business agility or work against it. At every level of the organization, the organization wants to structure itself to maintain the ability to rapidly change direction. Loose coupling between all elements within the enterprise is the key design factor for scaling agile.
Transparency
When an organization cannot get truthful information about how the project office is progressing against delivery, it is difficult to accurately anticipate change and respond to it. Agile teams strive for transparency through big visible information radiators that communicate team status in real-time to whomever is interested. The agile enterprise must have mechanisms in place for accurate information to become available to all decision makers, whether at the team level, the project and program level, or the enterprise portfolio management level.
Respects People
One of the primary differentiators between a traditional organization and the agile enterprise is the focus on respect for people. Respect for people in this context involves giving tremendous autonomy to people and teams to solve problems and be in control over their own work environment. This means that the organization can tell the team what to do, but the team decides how the work will get done and at what rate. The goal of the organization is to have a happy and healthy workforce that can operate at a sustainable pace indefinitely. The tradeoff for this individual and team autonomy is tremendous accountability and visibility into the product delivery life cycle, and demands that the individuals and teams become proficient at making and meeting commitments.
Enterprise Agile Design Considerations
Develop Agile Teams
Teams are the fundamental building blocks of agile organizations. There is not much in any agile method that works without the ability to create independent cross-functional teams that have everything they need to deliver and increment of business value. The problem with this concept is that some practitioners take it too far. Sometimes it is not possible to pull everyone required to create an end-to-end slice of the product into a single team. Sometimes programs are too large and too complex and some specialization is required to maintain system integrity. In these cases, the agile team must be organized around a set of features, a set of components, services, or business objects that have discrete business value irrespective of the rest of the organization. It will be the role of program management to pull together these discrete deliverables into a greater whole, and manage their production in a way that creates program level value as quickly as possible.
Requirements Management
Agile requirements management involves understanding at all levels of the relationship between the lower order requirements and the higher order requirements, all the way up to the portfolio level vision. The agile enterprise understands that requirements are going to change, but seeks to encapsulate lower level changes from impacting higher-level changes. By focusing on business value at all times, the enterprise is able to fine-tune the lower level requirements in such a way that allows each team to deliver the smallest increment of working product that will constitute a viable product deliverable.
Systems Thinking
The agile enterprise understands that the delivery of a project, program, or portfolio is a complex, interdependent system where all the pieces must work together and achieve balance across the whole. All too often, organizations optimize for one part of the organization at the expense of the entire system of creating value. This may manifest itself by grouping specialties together rather than focusing on complete agile teams. It may manifest itself through bringing in consultants to help with project management processes, when the reality is that it is the interplay between project management, requirements, and solutions delivery that is at the root of the issues. The agile enterprise focuses more on stabilizing the system and then managing how the projects flow through the system. This is often counter to the idea of stabilizing the portfolio and staffing to meet project demands.
Manage Constraints
Once the system of product creation is established and stable, constraints will arise that limit the ability to create value. In many product organizations today, constraints are ignored, or simply not managed, and bottlenecks are allowed to remain that restrict the organizations capacity to create value. Only by aggressively managing the system to help remove bottlenecks does the overall capacity of the entire system increase. No number of point solutions, unless applied to a bottleneck situation will increase the capacity of the organization to deliver.
Create Flow
The goal of the agile enterprise is not to deliver projects, but to increase the flow of value created by the organization. At every level of the organization, the focus is on delivering the highest priority requirements and features into the product. The agile project team is focused on delivering user stories into their component of the system in balance with the other teams building user stories into their components. The agile program management team is focused on delivering complete features across teams and balancing each team’s backlog such that they are working on the highest value program goals at any one time. The agile portfolio management team is focused on approving smaller initiatives and balancing the capacity of the organization to optimize the flow of value at the enterprise level.
Agile Project, Program, and Portfolio Execution at Scale
Portfolio Vision
The Portfolio Vision links the strategy of the organization into relatively small investment increments that can be quickly executed, proven, and introduced to customers on a regular cadence. These investment increments are called Epics and live at the top level of the organizational planning process. The Portfolio Vision allows the organization to understand how these Epics help the organization deliver against its broader set of business objectives. It includes information such as expected ROI, primary customers, competitive differentiation, market positioning, and competitive alternatives.
Story Mapping
Once the Epics have been identified, they are broken down into Features that can be delivered by the Program teams. Program teams shepherd these features through the Product Delivery life cycle, decomposing them into User Stories that can be delivered by a single team in an individual sprint. As teams deliver User Stories sprint over sprint, these User Stories are rolled-up across teams into completed features, which are ultimately rolled-up into completed Epics that support the business goals and can be measured for effectiveness. As markets change, Epics are discrete enough to be pulled out of development with minimal impact. As product needs change, Features are discrete enough to be pulled out of development with minimal impact. As technology and designs change, User Stories are discrete enough to be pulled out of development with minimal impact. User stories can change within the context of a Feature; Features can change within the context of an Epic, allowing the organization to inspect and adapt while still optimizing its top-level goals (Exhibit 1).
Exhibit 1 – Story Maps
Minimally Marketable Features
In the hierarchy of Vision to Epic to Feature to User Story, the minimally marketable feature represents the smallest slice of product functionality that still allows the organization to meet its market objectives with the product. Each layer of the governance should understand what constitutes the minimally viable product and manage scope in such a way that allows scope to change while minimally impacting top-level business goals (Exhibit 2).
Exhibit 2 – Minimally Marketable Features
Scrum at the Team Level
From an execution perspective, User Stories are sent to the team, via the Product Owner. The team works off a prioritized product backlog that is in sync and perfectly integrated with the backlogs of each of the other services or component teams. Each team is a self-contained entity that is responsible for delivering its increment of business value back to the program level organization. From this perspective, traditional Scrum is the management and planning metaphor. The teams work together to solve problems within the acceptance criteria contained within their backlog. They inspect and adapt, and as they learn more about the emerging product, the Product Owner brings that back to the program team to execute any mid-course corrections, and balance the changes in one team with the changes across the other teams.
Kanban at the Program and Portfolio Level
A two-tier Kanban model governs the flow of features at the team level, and the direction that the team receives from the Product Owner on a sprint-to-sprint basis. Epics flow through the top-level process flow. The Portfolio level team governs that rate of flow by making smaller investments, limiting work in process, managing capacity against demand, and identifying the removing bottlenecks from the system. Features flow through the second tier of the Kanban in a similar fashion, with the program level team governing the rate of flow at the team level, breaking down User Stories, and feeding the backlogs of each of the teams working across the program. From the team’s point of view, it is simply Scrum. At the Enterprise, the flow of value is made visible and managed by implementing the two-tier Kanban system (Exhibits 3 and 4).
Exhibit 3 – Three-Tier Planning Board
Exhibit 4 – Flow of Value Through a Three-Tier System
Conclusion
Managing and end-to-end agile enterprise requires a pragmatic approach to agile tools and techniques, and a willingness to apply the principles of agile and the Agile Manifesto in new and creative ways. As project managers, program managers, and portfolio managers, it is important to recognize that agile is not just a new way to writing software; it is a new way of conducting business. Team level agility without proper integration with the rest of the organization rarely yields the business benefits people expect. Rather than bend agile to the existing organization, the organization can recognize the benefits of agile at scale by adopting a more agile approach to portfolio and program governance.
