Agile software development and the space shuttle LiOH lab
Stephen M. Schneider
W. Don Gottwald
The Lithium Hydroxide (LiOH) Lab created Space Shuttle air purification cartridges used during shuttle flight (see Exhibit 1). These air purification cartridges were a mission-critical flight element, and one of the few mission-critical systems that had no backup. Without the LiOH air purification cartridges in active use during shuttle flight, the astronauts would quickly suffocate. If handled improperly, each LiOH cartridge had the potential to terminate both the mission and the lives of everyone aboard.
The LiOH cartridges were produced as a high-technology refurbishment. Used cartridges from previous shuttle missions were returned to the LiOH Lab, emptied, disassembled, cleaned, and reassembled with appropriate chemicals for future use. On earth, every processing step and all chemical consumables had to be tracked carefully and reported to NASA. The Cartridge Automated Resource Tracking (CART) software, developed using agile techniques, was used to track and report all LiOH Lab cartridge production.
Exhibit 1: STS-40 pilot Sid Gutierrez installing a LiOH cartridge during flight.
The LiOH Lab was a highly secure area, and LiOH cartridges were assembled in a controlled glovebox environment. Only six people had keys and the authority to be in the LiOH Lab unescorted. The multi-chamber glovebox you see in Exhibit 2 was pressurized with breathing air, which is normal air with no humidity. Nitrogen was originally proposed, but abandoned for safety reasons. A huge tank behind the LiOH lab contained the breathing air that kept the processing segments pressurized, so no ambient (and possibly contaminated) air could enter the box.
If you light a match in a room, the smell soon dissipates because of normal air exchange. However, in the space shuttle, you would continue to breathe the fumes, which would never dissipate. Because there is no air exchange when the shuttle is in flight, any poisonous or dangerous fumes and gasses must be removed from the flight deck atmosphere with one of the four different types of air purification cartridges.
Exhibit 2: LiOH cartridge glovebox processing stations.
By far, the most commonly used cartridge was the CO2 Absorber, or LiOH cartridge. This removed carbon dioxide produced by the astronauts. From 40-60 LiOH cartridges were flown with each shuttle flight, depending on the mission length.
The Agile Approach
The LiOH cartridges were critical flight hardware, and every step of production and all consumables had to be carefully tracked. In addition to simply gathering information, the tracking software also had to generate reports for different management levels and government agencies.
Management reviewed the tracking and reporting requirements and, using standard estimation practices, expected the LiOH Lab production tracking software effort to take 18 months. Fortunately, the LiOH Lab was more than 10 miles away from the main office. This relatively remote and highly secure location allowed us to use an agile software development approach (instead of the standard waterfall methodology) simply because we would not be under direct management supervision and were free to try a “radical” agile process.
Agile is very effective, and has increased in popularity, but it is not new. In 1962, John Paup was a senior NASA manager planning part of the Apollo program:
First thing every morning, all key people reported to his office for a 15-minute stand-up confessional in which they were to describe their major problem and explain what they planned to do about it in the next 24 hours…nobody was allowed to sit down (Gray, 1992, p. 106).
This is a very accurate description of a daily scrum in 1962! The concepts behind agile project approaches are not new, and they are not highly revolutionary. Nevertheless, they can be extraordinarily effective, and have been successfully used by a subset of project managers for generations.
In February 2001, a meeting of 17 software methodology experts defined The Agile Manifesto for Agile Software Development:
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
- Individuals and interactions over process and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
That is, while we value the items on the right, we value the items on the left more (Highsmith, 2002, p. xvii).
The response to this Manifesto was sudden and dramatic. According to Cockburn (2007):
The agile model of software development took the world by storm in 2001. Within a year there were books and conferences on it around the world. Within five years, it had influenced everything from project management and how corporate executives write contracts with their clients, to military procurement procedures, and even college curricula (p. xxxi).
With this growing popularity in mind, here are the individual components of the Agile Manifesto:
Individuals and Interactions Over Process and Tools
While processes and tools are certainly important, they do not perform the actual work. Individuals and interactions, the people involved and the teamwork created, are what makes things actually happen. If you want your department to produce a novel, which would be more important: the latest word processing software, or someone with a good idea and a desire to write? Too many organizations believe that all you need are good processes and tools, and the work will somehow do itself. If only life were so easy!
If you have watched the popular TV series NCIS, image Agent Gibbs saying to his superiors, “We need a new agent. Since we have good tools and processes already in place, anyone can join my team.” It would never happen.
Exhibit 3 shows the Death Star from George Lucas's first Star Wars movie. The Death Star, designed and built by a mighty empire, had a design flaw that the plans could not change. A small, talented group of rebels (a bunch of former nerf herders) led by Luke Skywalker used the force of agile to destroy the crowning technological achievement of an evil empire. Now that is what we call “individuals and interactions over process and tools.”
Exhibit 3: Death star from Star Wars.
Working Software Over Comprehensive Documentation
Make no mistake, documentation is important. Nevertheless, giving someone an excellent book on how to use Microsoft Office, but never providing the actual software, would be pointless. The goal of a software project is to deliver high-quality software. That software does not exist to support the documentation. The documentation exists to support the software. Some organizations confuse which of the two is most important. Management may be more interested in following the rules (which makes them look good to their superiors) and having a feeling of total control (which they find emotionally satisfying) than in actually getting the job done. Agile exists to create a working product. Documentation is important, but it is never as important as the working product itself.
Exhibit 4: DILBERT © 2007 Scott Adams. Used by permission of UNIVERSAL UCLICK. All rights reserved.
A “Dilbert” cartoon (Exhibit 4) shows the pointy-haired boss, Wally, and Dilbert sitting at a table. The pointy-haired boss says, “We're going to try something called agile programming. That means no more planning and no more documentation. Just start writing code and complaining. Wally replies, “I'm glad it has a name.” The pointy-haired boss retorts, “That was your training,” while Dilbert looks on silently as if nothing unusual has happened (Adams, 2007). This may be a popular perception, but it is not the true nature of agile.
Customer Collaboration Over Contract Negotiation
Written contracts have been with us since the Babylonians. Exhibit 5 shows a Sumerian loan contract. The collateral for the loan was the borrower's son. Those Sumerians took their contracts seriously! Contracts would not still be here 4,000 years later if they were not important. However, it is simply impossible for a contract to cover every contingency, every change in the environment, and every new technology that might possibly occur. As a result, agile demands a certain level of trust, so that both the customer and the seller can cooperate to manage inevitable change.
Exhibit 5: Sumerian loan contract.
Trust is not always easy to achieve, but it is in the best interest of everyone involved. Customer collaboration makes the customer a partner in solving the problem, while contract negotiation suggests a more adversarial relationship. Surowiecki (2004) observes,
It's impossible for a society to rely on law alone to make sure citizens act honestly and responsibly. And it's impossible for any organization to rely on contracts alone to make sure that its managers and workers live up to their obligations. So cooperation typically makes everyone better off. (p. 116)
Surowiecki (2004) also notes, “Capitalism is healthiest when people believe that the long-term benefits of fair dealing outweigh the short-term benefits of sharp dealing” (p. 126). For example, some American automotive companies focused on maximizing next quarter's profits (an example of short-term thinking) until they were bankrupt, while other automotive companies, such as Toyota and Honda, tended to take a much longer-term view and have profited as a result.
Responding to Change Over Following a Plan
Human beings have been working on the concepts of project management since the pyramids were built. Exhibit 6 shows the famous “bent pyramid,” designed about 4,600 years ago. After the pyramid was half-constructed, the engineers ran into problems. The base would not support the weight of all the stones planned for the upper portion of the pyramid. As a result, the architects changed the angle of the upper half to make it “flatter,” and, thus, use fewer stones. The result looks very awkward, and the upper part of the pyramid looks squashed. The change must have been embarrassing at the time, but the structure has stood the test of time. If the builders had kept to the original plan, the pyramid would have collapsed thousands of years ago.
Exhibit 6: The bent pyramid.
Orville and Wilbur Wright provide another example of responding to change over following a plan. They started their glider airframe designs in keeping with published data; however, much of the published information available on aeronautics was simply not correct (McCullough, 2015). Orville and Wilbur constantly adjusted their approach in response to their own findings. This was a highly agile approach that solved a problem that had challenged human beings since the days of Icarus.
One should also remember that the Wright brothers, “had no college education, no formal technical training, no experience working with anyone other than themselves, no friends in high places, no financial backers, no government subsidies, and little money of their own” (McCullough, 2015, p. 35). Nevertheless, their agile approach conquered heavier-than-air flight on an isolated, windswept beach, living in a handmade shack without electricity or running water. Such is the power of agile.
Plans are always important, but your plans must be able to change. This is not a new idea. Publilius Syrus, a contemporary of Julius Caesar, wrote: “Malus est consilium quod mutari non potest” or “Wicked is the plan that cannot change.” For example, the original Constitution of the United States made the person with the highest number of electoral votes president, and the person with the second highest number vice president. There was no combination president/vice president “ticket.” If the Constitution had not been amended, in the 2012 presidential election Barack Obama would have been elected president and Mitt Romney would have been elected vice president. Understandably, that may not have worked.
Not everyone supports taking an agile approach, and on occasion, agile is not appropriate. However, there is no doubt that those opposed to the fundamental philosophy of agile use those occasional instances as an excuse to dismiss agile in its entirety. This supports Wilson (1998), who observed, “Old beliefs die hard even when demonstrably false” (p. 256). This was certainly the situation with our former management, who were not familiar with agile and would not have supported a “passing fad” agile approach. At the LiOH Lab, we learned that Admiral Grace Hopper was right, “It's easier to ask forgiveness than it is to get permission,” a quote that was on the cover of the U.S. Navy's Chips Ahoy magazine in July 1986. In this instance, we had to explain, almost apologetically and in detail, why an agile approach delivered the LiOH Lab software one year early.
Harari (2002) summarized a rational response to agile opposition, “And, by the way, if your organization relentlessly smothers your efforts to initiate positive change, then you're probably on a sinking ship anyway. In that case, do what a lot of good players do: Polish your resume and look for the first opportunity to get out” (p. 75). This would have been exceptionally wise advice in light of the end of space shuttle program in 2011. We were on a sinking ship without knowing it. Our management's opposition to agile, once again, supported Wilson (1998), who noted, “We are drowning in information, while starving for wisdom” (p. 269).
The original estimated timeline for the LiOH Lab Cartridge Automated Resource Tracking effort was 18 months. After six months, management decided to check up on what we had accomplished. They expected that we would be finished with the preliminary requirements documentation and they wanted to see the book before we began coding. Upper management appeared quite shocked when we told them the project was completed. Apparently, they thought that their active input and guidance was needed for success.
The Agile Manifesto examined here is the foundation for Twelve Agile Guiding Principles. Together, the Manifesto and the Guiding Principles demonstrate that agile is simple, but not easy. Management (and employees) must make a fundamental change in their worldview, which can be emotionally challenging and personally uncomfortable. However, the rewards provided by agile software development are so dramatic, and the long-term benefits so advantageous, that agile is well worth almost any emotional and personal investment. It is now up to us. At the LiOH Lab, agile software development techniques strengthened the safety, reliability, and availability of a critical space shuttle flight element.
Adams, S. (2007, November 26). Dilbert [Cartoon strip]. Retrieved from http://dilbert.com/strip/2007-11-26
Agile Alliance. (2001). The agile manifesto for agile software development. Retrieved from: http://agilemanifesto.org/
Cockburn, A. (2007). Agile software development: The cooperative game (2nd ed.). Upper Saddle River, NJ: Addison-Wesley.
Gray, M. (1992). Angle of attack. New York, NY: W.W. Norton.
Harari, O. (2002). The leadership secrets of Colin Powell. New York, NY: McGraw-Hill.
Highsmith, J. (2002). Agile software development ecosystems. Boston, MA: Addison-Wesley.
McCullough, D. (2015). The Wright brothers. New York, NY: Simon & Schuster.
Surowiecki, J. (2004). The wisdom of crowds. New York, NY: Doubleday.
Wilson, E. O. (1998). Consilience: The unity of knowledge. New York, NY: Alfred A. Knopf.
© 2015, Stephen M. Schneider and W. Don Gottwald
Originally published as a part of the 2015 PMI Global Congress Proceedings – Orlando, Florida, USA