delivering business value via the nexus of lean-agile and IT service management
Dr. Ahmed Sidky, Principal Consultant, Sidky Consulting Group
The delivery of business value is a seminal function that can and should drive decision-making associated with IT portfolio management and service delivery. All too often, however, those working in software development find themselves at odds with staff working in IT operations. The good news is that there have been some encouraging trends and significant success stories in which software development and operations functions are finding ways to collaborate more effectively, even in large and complex organizational contexts. In fact, even though it might seem that the need to react rapidly to shifting business priorities, which are often manifested via frequent software deployments, runs directly counter to the delivery of stable and predictable IT services, a growing number of organizations are proving that development and operations functions can partner effectively.
This paper seeks to show how lean-agile development, which is increasingly common in the software development community, is at its core, driven by values and principles, along with a customizable set of practices, which align very nicely with the principles and practices of IT Service Management (ITSM) and the IT Infrastructure Library (ITIL), which are so prevalent among operations practitioners. The authors of this paper are thus coining the term “ITILity” to represent the nexus of agility and ITSM/ITIL. Finally, and most importantly, this paper provides insight into how IT can help organizations achieve the business outcomes they wish to achieve, by leveraging the best that lean-agile software development and ITSM have to offer.
Examining the Sources of Conflict between Software Development and IT Operations
The relationship between software development teams, which focus on creating and modifying software, and IT operations groups, which have responsibilities that include deployment of software packages to production environments is often contentious. For instance, to look at things from an IT operations perspective, operations teams are evaluated largely on the extent to which the systems they manage are stable. Given this fact, it is understandable that there is a desire to restrict deployment of software to a production environment as much as possible, to avoid the instability that is so often associated with software releases (Humble & Molesky, 2011, p. 6). Organizations are discovering ways to make the relationship between software development (“Dev”) and operations (“Ops”) much less contentious, and a term that is often used to describe this evolving relationship is DevOps. This paper seeks to provide further evidence that it is not only possible to bridge the perceived divide between the Dev and Ops communities, but also to show how their shared interest in the delivery of business value makes Dev and Ops natural allies.
One of the changes in the IT industry that is prompting organizations to change their perspective on the DevOps relationship is a shift from traditional software development approaches, based on a phase-based, or “waterfall” model, to agile software development. In sharp contrast to waterfall development, which for larger implementations tends to produce working software relatively infrequently, agile software development makes it possible to deliver working software much more rapidly, and at a predictable cadence. Although from a business perspective the rapid delivery of software may seem like a good problem to have, from a technical perspective, frequent deployments of software to production environments find many organizations unprepared for such a profound shift in their internal operations.
“A number of people got together and started a grass-roots movement that set out to remove the traditional boundaries between development and operations. Some consider this picking up where ‘traditional agile’ left off. After all, software doesn't bring value unless it is deployed in production; otherwise, it's just inventory. To tackle the problem, devops encourages cross-silo collaboration constantly, not only when things fail.” (Debois, August 2011, p. 3)
An Introduction to Agile, Lean, and IT Service Management
This section introduces the mindset, values, principles, and practices associated with agile software development, lean software development, and IT Service Management, respectively. Before introducing each of these three approaches, it is important to first distinguish between principles and practices. Principles “…are underlying truths that don't change,” while practices “…are the application of principles to a particular situation” (Poppendieck & Poppendieck, 2007, p. 19). The Poppendiecks go on to suggest that a learn-by-doing approach is very important, in that applying a particular set of practices helps organizations better understand the underlying principles (along with the mindset and values) behind those principles.
To state the difference between principles and practices in more familiar terms, principles focus on why an organization does something, whereas practices focus on how an organization takes action to realize those principles. Far too often, organizations focus the majority of their energy on how they are doing something, and lose sight of why they are doing what they are doing. Exhibit 1 illustrates the relationship among Agile Values, Principles, and Practices (Smith & Sidky, 2009, p. 6).
As with so many areas of human endeavor, the key is to strike a balance between the why and the how, between the strategy (values and principles) and the execution (practices). It should not come as a surprise, therefore, that the most successful organizations are those able to find such a balance.
Overview of Agile Software Development
One of the most common impediments that can cause an organization to not be able to realize the full potential of agile software development is what many refer to as the difference between “doing agile” versus “being agile.” An interesting way to look at the difference between doing agile and being agile is to see the value of a project as why divided by how (value = why/how). Suppose that an organization has adopted a variety of agile practices, without taking the time to articulate a vision regarding what they expect to gain by adopting those practices. Since it is hard to argue that the numerator (the why) in such a situation is a non-zero value, no matter how many practices the organization might adopt, they are unlikely to realize any significant gains from their adoption of those agile practices, because zero divided by ten is no different than zero divided by fifty (Hanoulle & Linders, 2012, ¶3).
One might argue that representing complex organizational dynamics in terms of such a simple mathematical formula fails to take into account the many factors that are at work in such situations. To make such an argument is to miss the point—that organizations focusing excessively on the how are doing agile, without being agile. In other words, such organizations fail to manifest an agile mindset. As Steve Vaughn points out (2013, ¶1), the agile mindset needs to be central to an organization's culture. The centrality of the importance (and difficulty) associated with organizational change is borne out by the results from VersionOne's State of Agile Development Survey, which in addition to providing data about the many agile success stories, also shows that the most significant barrier to further agile adoption is the ability to change organizational culture (VersionOne, 2013, p. 7).
Returning to the DevOps relationship, which is so central to this paper, for organizations considering an agile transformation consider the difference in terms of the likelihood of success between an organization that takes the time to do a proper assessment of its culture, its values, and its principles, versus an organization that makes no such effort. In the former case, it would be typical for a broad spectrum of stakeholders to be involved in the planning, design, implementation, and continuous improvement of the agile transformation initiative, and those stakeholders would most certainly include individuals representing various business functions, along with staff working in software development and IT operations roles. By way of contrast, in the latter case, where an organization rushes into an agile transformation, it is all too frequent an occurrence that key stakeholders, including those in IT operations, are not adequately consulted, often with disastrous consequences. To put it in more concrete business terms, “IT organizations that fail to confront and reconcile the widening gap between their development and operations teams stand to lose their footing in today's competitive business environment” (Orr, 2012, p. 1). The fact is, in organizations where an agile transformation not only survives but thrives, the agile mindset, values, and principles span the entire organizational value stream, from concept to cash.
Exhibit 2 shows the nature of the cyclical relationship among business stakeholders (application owners), the software development function, and the IT operations function.
This simple diagram is also a good metaphorical representation of the four agile values, as articulated in the Agile Manifesto (Beck et al., 2001). The four values are: Individuals and interactions over processes and tools; Working software over comprehensive documentation; Customer collaboration over contract negotiation; and Responding to change over following a plan. Imagine how well such a cycle would work if there were insufficient interaction and collaboration among the parties involved, or if they did not all see the delivery of working software as important. These four values, along with twelve principles (which constitute the second part of the Agile Manifesto), must be at the very core of every agile initiative, regardless of which set of practices an organization decides to adopt. In other words, an agile mindset is needed for an organization to be agile, as opposed to just doing some agile practices without the accompanying mindset, values, and principles.
Overview of Lean Principles and How They Apply to Software Development and IT Operations
Just as with agile software development, lean software development too places great importance on the values and principles of an organization. “Principles drive behavior and behavior determines results. Most change programs start from actions and the results begin and end there. But if you want sustainable results, the only way is to get principles and behaviors right. Sustainable lean change has to work at the principles and behaviors level” Flinchbaugh, 2012, p. 17).
One of the lean tools that organizations sometimes use is called “the five S's.” The S's represent five Japanese words that start with an S. Exhibit 3 shows an interesting application of the five S's to Java software, based on an email exchange between Kent Schnaith and Mary and Tom Poppendieck (Poppendieck & Poppendieck, p. 192).
Shifting focus to another Lean principle, a concept that is relevant to those working in both software development and IT operations, is the need to keep batch sizes as small as possible. To start with a reasonably simple definition, batch size is synonymous with the size of a software module or similar work product that moves from one environment to another. Thus, every time a developer checks in code, they are batching a certain amount of work. Lean-agile software development employs various practices to keep batch size reasonably small, such as Continuous Integration (CI), where relatively small sets of changes are run against a set of tests that ensure build integrity is maintained. Some of the benefits of smaller batch size and CI include faster feedback to developers, keeping any problems that do occur localized, and reduced code integration risk, all of which benefit parties involved with software development and deployment (Ries, 2009, ¶ 2–7).
Stated slightly differently, CI is an excellent illustration of how lean and agile concepts, when used in combination, can produce superb results. And, CI can benefit both software development and IT operations staff, in the former case by catching defects early, along with making the build process more robust, and in the latter case by making it easier to migrate software from one environment to another in a much more predictable fashion. “In lean thinking terminology, CI replaces big batches and long cycle times of integration (the practice of traditional configuration management) with small batches and short cycles of integration—a repeating lean theme” (Larman & Vodde, 2008, p. 182).
To sum up this short section on lean software development, as with agile, there are many lean success stories. And, just as it is with agile, successful lean implementations tend to follow certain patterns. Lean inherits ideas from the world of manufacturing to drive efficiency and effectiveness, ideas that are particularly relevant in IT operations. Lean ideas can work very nicely in tandem with agile, given its strong alignment with the thought processes and work patterns associated with software development. “What is really behind companies that succeed at sustained lean implementation is the level of thinking driven by lean principles and rules. Thinking is powerful in changing an organization. Thinking drives behaviors. Behaviors drive action. Action drives results.” (Flinchbaugh, 2012, p. 4)
Overview of ITSM and ITIL
At its most basic level, the main purpose of ITSM is to make sure that IT services both align with and can reliably support the business needs of an organization. As stated in the IT Service Management Forum (itSMF) introductory guide to ITIL Version 3 (Cartlidge et al., 2007, p. 5), “It is imperative that the IT services underpin the business processes, but it is also increasingly important that IT acts as an agent for change to facilitate business transformation.” ITIL is the most common wrapper, or framework, for ITSM. ITIL seeks to make it possible to continually measure and improve the quality of the IT services that are delivered.
ITIL Version 3 consists of five core books that are intended to represent the five major stages of the life cycle of an IT Service, as shown in Exhibit 4. The life cycle begins with definition of the service and analysis of business requirements (during the first two stages, service strategy and service design), continues with migration into a production environment (the service transition stage), and then on to service operation, support, and continuous improvement (which are the areas of focus during the service operation and continual service improvement stages).
To close this short section on ITSM and ITIL, two primary characteristics that are associated with IT services — fit for purpose and fit for use — are tremendously important in both lean-agile software development circles and among IT operations staff.
“[An IT service]…should be fit for purpose — in other words, it should deliver the expected value to users in terms of functionality, and it should be fit for use, which means it should be production-ready in terms of meeting capacity, availability, and security characteristics. Poor collaboration between developers, testers and users results in software that is not fit for purpose. Poor collaboration between developers, testers and operations leads to software that is not fit for use. Bureaucratic change management processes that are put in place to reduce the risk of releases often create silos that further inhibit collaboration. Delivering software is an inherently collaborative exercise, so it should come as no surprise that effective release management of software that is both valuable and production-ready requires organizational change to enable collaboration between everybody involved in delivery.” (Humble, 2010, pp. 2–3)
Comparing ITSM Activities with Lean-Agile Activities
To set the stage for a comparison of ITSM processes and practices with lean-agile practices, here is a brief review of lean, agile, and ITSM values and principles: (1) Agile begins with a mindset, reinforced by a core set of values and principles, along with a customizable set of practices, where a customer is actively involved with software development teams that produce releasable, valuable software on a frequent basis; (2) Lean places emphasis on optimizing end-to-end processes to create customer value, with particular focus on the flow of work product, along with removing bottlenecks wherever they exist in a process, and streamlining wasteful activities, and; (3) ITSM enables IT service providers (IT operations in particular) to align the IT services they provide with the organization's business objectives and thereby work with their business partners to define parameters such as critical success factors (CSFs) and key performance indicators (KPIs) associated with each IT service.
Having touched upon the key values and principles associated with lean, agile, and ITSM, a shift in focus to the practical application of these values and principles via processes, activities, and practices is now in order. Exhibit 5 compares the key processes and activities associated with ITIL service life cycle stages with lean-agile practices and activities. Process and activities associated with the following three ITIL service life cycle stages are included in the table*:
- Service Transition. The IT service life cycle phase that begins with the deployment of Configuration Items (CIs), such as code, configuration files, and scripts, to one or more pre-production environment, and ends with the deployment of those CIs into the production environment.
- Service Operation. The IT service life cycle phase that focuses on ensuring that IT services conform to Service Level Agreements (SLAs) and/or other types of agreements as agreed on with stakeholders.
- Continual Service Improvement. The IT service life cycle phase that focuses on making ongoing improvements, both for IT services as well as the processes underlying the ITSM service life cycle.
* Three of the five ITIL service life cycle stages are included in the table because it is these three stages that focus on the transition of a service to a production environment, the support of that service while it is in production, and the continuing need for an organization to identify opportunities for improvement in these and other process areas.
DevOps Patterns and Anti-Patterns
Exhibit 6 describes things that have been proven to work (patterns) and things that experience has shown do not work (anti-patterns) in a DevOps context. In other words, the patterns represent behaviors that contribute to a healthy DevOps relationship.
* Duvall, 2010
There is no better illustration of the importance of the Dev-Ops relationship to the health and well-being of an organization than the story that is told in The Phoenix Project (Kim, Behr, & Spafford, 2013). The story begins with an organization that is clearly in trouble, beset with financial problems and organizational dysfunction, with a high degree of conflict among business and IT stakeholders. Given this backdrop, a mid-level IT manager suddenly finds himself in very senior position where he is given a mandate (more of an ultimatum, really) to address many of the organization's most dire problems. During his early days in the new position he discovers just how central a healthy Dev-Ops relationship is to being able to make the sort of improvements he has been asked to make. Kim, Behr, and Stafford not only tell a fascinating story, they describe realities that they have observed in many different organizations, and even more importantly, they illustrate that it truly is possible for an organization to realize the benefits of lean-agile software development along with ITSM, and to build a healthy, collaborative relationship among those in software development and those in IT operations that works to the benefit of the organization as a whole.
When it comes to realizing the benefits of lean-agile software development, some of the more obvious benefits include faster development of features and an environment which fosters innovation. But it is important not to lose sight of the most important benefit, which is the ability to rapidly deliver business value, in whatever form business value might be realized for an organization. When viewed from the perspective of one of the most critical manifestations of business value delivery — competitive advantage — organizations that practice lean-agile software development and that “…have the ability to quickly and inexpensively evolve a product closest to the end of the development lifecycle will have a tremendous competitive advantage. Ultimate customer value is delivered at the point-of-sale, not the point-of-plan” (Highsmith, 2010, p. 8). It is well within the grasp of any organization to build a foundation of collaboration and trust that enables those in software development to partner with those in IT operations to the mutual benefit of everyone in the organization.
In conclusion, although there are numerous examples of contentious DevOps relationships, there is no reason that teams pursuing agile software development inherently have to be in conflict with IT operations. “To successfully create the significant breakthroughs in your development effectiveness that are possible with agile, it has to be aligned with why you want to do it in the first place and what you need to achieve from it. You should be agile not just to be agile, but to drive the business results.” (Gruver, Young, & Fulghum, 2012, p. 9)
Beck, K., et al. (2001). The Manifesto for Agile Software Development [Online]. Retrieved from http://www.agilealliance.org/the-alliance/the-agile-manifesto/.
Betz, C. (2007). Architecture and Patterns for IT Service Management, Resource Planning, and Governance: Making Shoes for the Cobbler's Children. San Francisco, CA: Morgan Kaufmann.
Cartlidge, A., et al. (2007). An Introductory Overview of ITIL V3, version 1.0 [Online]. Retrieved from http://www.itsmfi.org/files/itSMF_ITILV3_Intro_Overview.pdf.
Debois, P. (August 2011). “Opening Statement,” Cutter IT Journal, vol. 11, no. 8, pp. 3–5 [Online]. Retrieved from http://nobelium.se/itj1108.pdf.
Duvall, P. (2010). “Continuous Delivery: Patterns and Antipatterns in the Software Lifecycle,” DZone refcard #145. Retrieved from http://refcardz.dzone.com/refcardz/continuous-delivery-patterns.
Gruver, G., Young, M., & Fulghum, P. (2012). A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware. Upper Saddle River, NJ: Addison-Wesley.
Hanoulle, Y., & Linders, B. (Dec. 28, 2012). “Interview with Yves Hanoulle on the Agile and Lean Mindset” [Online]. Retrieved from http://www.infoq.com/articles/agile-lean-mindset.
Highsmith, J. (2010). Agile Project Management: Creating Innovative Products, 2nd Edition. Upper Saddle River, NJ: Addison-Wesley.
Humble, J. (2010, 14 July). Agile Release Management: Towards Frequent, Low Risk Releases [Online]. Retrieved from http://www.kn-portal.com/fileadmin/xxx/AgileReleaseManagement-whitePaper.pdf.
Humble, J., & Molesky, J. (August 2011). “Continous Delivery,” Cutter IT Journal, vol. 11, no. 8, pp. 6-12 [Online]. Retrieved from http://nobelium.se/itj1108.pdf.
Kim, G., Behr, K., & Spafford, G. (2013). The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win. Portland, Oregon: IT Revolution Press.
Larman, C., & Vodde, B. (2008). Scaling Lean & Agile Development: Thinking and Organizational Tools for Large-scale Scrum. Upper Saddle River, NJ: Addison-Wesley.
Orr, A. (December 2012). “Maximize the Synergies between IT and DevOps” [Online]. Retrieved from http://documents.bmc.com/products/documents/52/40/435240/435240.pdf.
Poppendieck, M., & Poppendieck, T. (2007). Implementing Lean Software Development: From Concept to Cash. Upper Saddle River, NJ: Addison-Wesley.
Ries, E. (February 20, 2009). “Work in small batches,” Startup Lessons Learned [Online]. Retrieved from http://www.startuplessonslearned.com/2009/02/work-in-small-batches.html.
Smith, G. & Sidky, A. (2009). Becoming Agile in an Imperfect World. Greenwich, CT: Manning.
Vaughn, S. (May 31, 2013). “Putting into Practice an Agile Mindset” [Online]. Retrieved from https://www.techwell.com/2013/05/putting-practice-agile-mindset.
VersionOne (2013). “The Seventh Annual State of Agile Development Survey” [Online]. Retrieved from http://www.versionone.com/state-of-agile-survey-results/.
©2013 G. Philip Rogers & Dr. Ahmed Sidky
Originally published as part of the 2013 PMI Global Congress Proceedings – New Orleans, Louisiana