Government Software Teams Are Adopting Agile—And Shedding Bureaucratic Stereotypes
BY STEVE HENDERSHOT
PORTRAITS BY NOAH WILMAN
Kimberly Hancher, Deep Water Point, Annapolis, Maryland, USA
Agile approaches are no longer an enemy of the state. Following the private tech sector's lead, governments around the world are finally embracing change by adopting agile as a preferred delivery approach. While the transition is restricted primarily to software projects, the trend is helping government agencies shake their reputation for being slow-moving monoliths incapable of adapting to change.
National governments in the United States, United Kingdom and Australia have adopted guidelines that endorse agile for software projects. Countries ranging from Brazil to Singapore also are pushing to incorporate agile into more of their government IT projects.
In the U.K., for example, the Government Digital Service mandates a user-first approach to all IT projects across all government agencies. Last year it capped all its project lengths at 11 weeks—the scope can flex, but not the schedule. The shift in practices appears to be speeding up in the U.S.: Project leaders say 80 percent of current major federal IT projects are agile or iterative, up from just 10 percent in 2011, according to a 2017 Deloitte analysis. (Federal project management practices are evolving more broadly, too. The U.S. government is implementing the Program Management Improvement and Accountability Act, which was endorsed by PMI. Signed in December 2016, it develops project and program management standards and guidelines for the U.S. government.)
“Government project teams cannot choose a particular market. … So we have to make sure we understand how to design for all relevant users.”
—Rebecca Piazza, 18F, Washington, D.C., USA
The reasons for the public sector's trend toward agile delivery practices are obvious to agilistas: Software projects that use agile report higher success rates than their waterfall counterparts, and agile's focus on breaking large initiatives into small, manageable chunks can address the tendency toward cost and scope creep that has hobbled many government IT projects. (In 2004, for example, the average U.S. government software project lasted nine years and cost US$144 million. These days, it's eight months and less than US$2 million.)
Yet building government applications presents unique challenges that tech startups and corporate skunkworks often don't face. This includes time-consuming regulatory requirements that can impede agile's preferred rapid-fire pace. For instance, Kimberly Hancher, a former CIO of the U.S. Equal Employment Opportunity Commission (EEOC), points to a U.S. law mandating that any time a federal agency updates an official form, the document is subject to a three-stage review process that includes 90 days in which the public can comment on the change.
“It's just very time-consuming and difficult to coordinate with system builds,” says Ms. Hancher, now a principal consultant at Deep Water Point, Annapolis, Maryland, USA. “It's something that might have made sense when all information collections were on paper. But with an agile process, it's totally burdensome.”
Another challenge is scope. Government projects often must serve a broad user base—the general public—so pivoting toward a niche solution isn't an option.
“Unlike in the private sector, government project teams cannot choose a particular market or choose not to serve some market segments. So we have to make sure we understand how to design for all relevant users—and make that the focus of implementing policies and regulations,” says Rebecca Piazza, acting executive director of 18F, Washington, D.C., USA. The U.S. government-owned digital services agency was created in the wake of the country's botched healthcare.gov rollout and works with federal agencies to use agile on their IT projects.
To some extent, the tension between agile and government is healthy. Nobody wants nuclear-weapons scientists to adhere to the agile credo “fail fast,” for example. But the common tendency inside government agencies to believe that more specific regulations and more prescriptive policies lead to better outcomes is a problem, says Kirk Rieckhoff, partner, McKinsey & Co., Washington, D.C. He co-authored a 2017 report on how agile practices can benefit government. “That's completely antithetical to agile development,” he says. “Rather than over-defining what needs to get done, you have to allow room for the magic to happen.”
Portion of major U.S. federal IT projects that were agile or iterative in 2017
Portion of same type of projects described as such in 2011
Source: Deloitte Center for Government Insights, 2017
There are real signs, however, that two of the greatest cultural barriers to agile adoption in government—the approach's user focus and emphasis on quickly generating and testing working software—can be overcome.
Ms. Hancher was a recent agile convert when she saw an opportunity to shift away from waterfall at EEOC in 2015. The agency was embarking on a project to create an online portal through which employers accused of discrimination could exchange documents and track the status of their cases. Ms. Hancher's 20-person team had just five members with experience working on agile projects, so her first order of business was education. The whole team took three days of agile training, and two people (the project manager and the lead developer) were trained as scrum masters—chief facilitators of the short-term sprints that partly define the agile process. In addition, Ms. Hancher and the project manager visited other government agencies that had implemented agile to observe their processes.
“You need to have people who use the system day in and day out telling you in plain language what works well and what doesn't work well.”
Kimberly Hancher, center, with team members
“It was eye-opening for a lot of us who were used to focusing on things like documentation of requirements and change requests,” Ms. Hancher says. “It put the emphasis more on creating usable components and working closely with the product owner.”
The learning curve was steep, however. As the project got underway, it quickly became clear that a focus on customer use cases represented a new way of thinking for waterfall-trained developers.
“Government agencies traditionally have not been focused on external customers, and designing systems has been kind of painful because the product owners often don't have a good idea of what their customer really needs,” Ms. Hancher says.
So her team held focus-group sessions with representatives from companies that interact with the EEOC. It was a classic agile move, but it raised eyebrows within the agency.
“There was a lot of angst about involving these end users, but we needed them to give honest, direct feedback. You need to have people who use the system day in and day out telling you in plain language what works well and what doesn't work well,” Ms. Hancher says.
The quality of the feedback—and the resulting improvements—soon converted skeptics. Participants in one focus group, for example, expressed concern that the system would email notifications back to only the submitter's address. They suggested one simple change: route email notifications to multiple addresses at the company instead of to a single employee who might miss the message if sick, on vacation or separated from the company.
When Australia's National Blood Authority (NBA) decided to update its proprietary BloodNet software used to track and distribute blood through the country's healthcare system, its leadership was determined to use agile approaches. The agency wanted to align its delivery practices with a push for greater agility on major multiyear IT projects by the Australian government's Digital Transformation Agency.
Two-week sprints, standing meetings and feedback-driven iterations were all new experiences for the NBA development team as it developed the new version of BloodNet. “The concept of digital transformation and agile delivery within the organization was challenging at first. However, the team's mindset evolved over the course of the project,” says Sabeshan (Shane) Sithamparanathan, deputy CIO/program manager, NBA, Canberra, Australia. “Given we are now delivering the service to our end users, user-centered delivery is a core part of our work.”
Led by Mr. Sithamparanathan, the team conducted user research at 39 hospitals and laboratories in nine cities. Once it had a working prototype running at its offices in Canberra, it invited 30 users from across the country to come and test the product. Their feedback led to modifications to the way the system handles documentation related to order fulfillment. In another test, the NBA put 55-inch monitor screens in select labs to see if the displays made the system easier to use. They did—and the screens are now standard. The new version of BloodNet is currently in beta-testing. Approximately 28,000 liters (7,397 gallons) of blood are ordered and tracked through the platform each month. The system already is reducing blood waste by millions of dollars annually.
“It's entirely possible for government agencies to build their digital capabilities,” Mr. Sithamparanathan says. “Doing so requires bringing digital skills in-house and upskilling existing employees. It's exciting to watch teams adapt to agile principles, tools and techniques.”
“Agile is a substantial and nuanced change to the way you think about doing things. It's an organizational change.”
—Justin Warren, PivotNine, Melbourne, Australia
ALL OR NOTHING
The BloodNet project shows a team can successfully implement agile—even a team steeped mainly in waterfall delivery practices. But many agile advocates worry that government agencies will incorporate only bits and pieces of agile within a larger waterfall framework—a hybrid model that Ms. Piazza calls “agile-fall” and says could inhibit agile's effectiveness. Agilistas worry that if projects stall as a result of halfhearted implementation, agile itself will be blamed.
“Agile is a substantial and nuanced change to the way you think about doing things. It's an organizational change, not just a project management change,” says Justin Warren, managing director of IT consultancy PivotNine, Melbourne, Australia.
Agile-fall is “frustratingly common,” says Rob Rees, head of operations at the consultancy Box UK, London, England. “There are both structural and cultural challenges that must be addressed in order to enact major change.”
The good news is that organizational and culture changes are entirely possible—government and agile approaches aren't intrinsically at odds with each other. If agile can demonstrate its effectiveness—as it already has in many agencies—genuine adoption should keep gaining steam.
“A lot of the barriers to change are self-inflicted and can be cleared,” Mr. Rieckhoff says. PM
HELP WANTED—AND FOUND
Government agencies trying to make agile approaches the new delivery norm are on the hunt for agile talent. The good news is that—at least in the United States—luring agilistas into the public sector isn't a problem.
“We have many more qualified people applying than we have roles to fill,” says Rebecca Piazza, acting executive director of U.S. digital services agency 18F, Washington, D.C., USA. “There is a high level of enthusiasm for bringing agile to government work.”
Former U.S. Equal Opportunity Employment Commission CIO Kimberly Hancher, Annapolis, Maryland, USA, agrees, adding that agile implementations work best with new project teams. Reorienting an existing team is more challenging. It helps to seed the group with a few newcomers who arrive with agile expertise and enthusiasm, so that their zeal spreads to employees who might be more used to waterfall. The shift to agile, Ms. Hancher says, is “hard for software development professionals who prefer to work independently. If you don't like talking, sharing and transparency, then you won't like working on an agile team.”
“There is a high level of enthusiasm for bringing agile to government work.”