Disciplined Agile

Release Management Strategies for DevOps

There are four general release scheduling strategies that potentially support DevOps. These strategies, from least effective to most effective, are:

  1. Release windows (slow cadence). A release window is a period of time during which one or more teams may release into production. A release slot is subset within that release window (and may be the entire window) during which a team may deploy their solution into production. For example, your organization may have a policy that production releases occur between 1am and 5am on Saturday evenings (the release window) and that up to four releases may occur during that window (the release slots). In lean terms, a release slot is effectively a Kanban card that allows a team to deploy. Release windows tend to align with periods where system usage is lower, although in the modern world of global 24/7 operations these periods have all but disappeared. With a slow cadence approach to this strategy the release windows occur far apart, as seldom as once a week or even longer. The advantages of this approach are that it provides a consistent release cadence to business stakeholders and it provides consistent release date targets for delivery teams. The primary disadvantage with slow cadence release windows is that they become bottlenecks for release management processes that need to support multiple teams. There are only so many release slots available during each window and this number can be easily exceeded, forcing teams to aim for a future release window. This problem becomes exacerbated when teams start to move to a continuous delivery strategy.
  2. Release train. The idea with a release train is that every team involved with that “train” has the same release cadence — for example this train releases once a quarter, or once a month, or even once a week. This strategy is commonly used in large programs, or teams of teams, where the individual teams are each working on part of a larger whole. Having the common drumbeat of a release train provides a consistent release schedule for stakeholders and serves as a rallying point for development teams. The train metaphor works quite well in practice. If your team misses the release date, if you miss the train, then the train goes on without you and you need to wait for space on the next on. Dependencies are also respected. For example, if several components need to ship together they must all go on the same train (similar to a family taking a trip together). The primary disadvantage is that development teams are constrained to a common release schedule, making it difficult to support lean or continuous delivery strategies. A potential disadvantage is that release trains may also suffer from the bottleneck problems of slow cadence release windows.
  3. Release windows (quick cadence). To support continuous deployment, particularly across many delivery teams, you will need a large number of release slots. The implication is that you will also likely need more release windows more often. The advantage of quick cadence release windows is that they are less likely to suffer from the bottleneck challenges associated with slow cadence release windows and release trains.
  4. Continuous release availability. With this approach delivery teams are allowed to release their solutions into production whenever they need to. In many ways this is simply an extension of the release window strategy to be 24/7. This is the only strategy that truly supports continuous delivery. To make it work a host of DevOps practices are required, such as fully automated deployment, fully automated regression testing, feature toggles, self-recovering components, and many others are required.

Our experience is that most enterprises today employ a slow cadence release window approach although are starting to evolve into the quick cadence version of this strategy. This is usually motivated by the adoption of agile techniques by solution delivery teams and more often than not by continuous delivery practices. We also see large programs take a release train approach, a strategy pioneered in the 1990s by large software companies such as Microsoft and Rational that sold software suites comprised of many products that needed to be shipped together. In recent years the OpenUP and SAFe frameworks have popularized the release train strategy. The strategy of continuous release availability is commonly used in advanced DevOps organizations such as Etsy and Amazon.

In addition to the release scheduling strategies listed above, there are several technical release management strategies that support DevOps:

  • Integrated deployment planning. From the point of view of development teams, deployment planning has always required interaction with an organization’s operations and release management staff; in some cases, via liaison specialists within operations often called release engineers. When you adopt a Disciplined DevOps mindset, you quickly realize the need to take a cross-team approach to deployment planning due to the need for operations staff to work with all of your development teams. This isn’t news to operations staff, but it can be a surprise to development teams accustomed to working in their own, siloed environments (luckily this strategy is built into DAD’s Accelerate Value Delivery process goal). If your team is not doing this already, you will need to start vying for release slots in the overall organizational deployment schedule.
  • Standard development and testing environments based on production. Development teams know that the greater the consistency between their development, testing, and production environments the easier it is to test and deploy. In multi-team environments the implication is that this will result in de facto standardization of many aspects of your environments. Developers may choose different development tools, but aspects of the infrastructure such as operating systems, application servers, middleware, databases and so on will become consistent over time to streamline the overall release process.
  • Release service streams. A key tenant of DA is that every team is unique, and an implication of that is that some teams will need more help than others. Teams will produce different levels of quality, they will have different amounts of automation, they will have different release cadences, and so on. As a result your release management strategy needs to be flexible enough to address these different situations. One way to do so is to offer different server streams, or service levels as it were, to solution delivery teams. For example, you may have a basic release management service stream where release management engineers actively help delivery teams to deploy their solutions into production and even help them to start automating some of their processes. At the other end of the spectrum you may have a continuous delivery service stream for delivery teams that have fully automated their testing and deployment processes and that can be trusted to successfully deploy on their own. And of course you could have several other service streams between those two extremes. The advantage of this approach is that it is very flexible albeit at the cost of slightly greater scheduling complexity.
  • Release blackout periods. Some organizations have periods of time where they choose not to release new functionality into production unless it is absolutely critical. These blackout periods typically occur during high-volumes of business transactions. For example, many North American retail companies will have blackout periods between mid-November and early January for the holiday sales seasons. Many organizations will have blackout periods near the end of their fiscal years to enable them to focus on the process required to close out the year.
  • Shared release practices. Although this is really a process improvement issue, it’s worthwhile to point out that whoever is involved with release management should actively strive to share effective practices between teams. Sharing learnings across teams is an important aspect of enterprise awareness.

Release management is an important part of your Disciplined DevOps strategy. Having said that, many IT departments are still in their early days of adopting a DevOps approach yet still effective release management. The implication is that the way that you approach release management will vary depending on how far down the DevOps adoption path you are. For example, with no DevOps in place at all your release management activities are likely to be performed by a team that is separate from your delivery teams. When you are in the process of adopting a DevOps mindset , release management is likely to be a collaborative effort between the delivery teams and the release management team. When you have fully adopted DevOps strategies release management is mostly performed by the delivery teams themselves.