Escaping Pilot Purgatory

Chasing Innovation Isn't Enough; Organizations Need Project Management To Make Test Projects Pay Off


Shailendra Malik, Optimum Solutions, Singapore



Pilot projects can bridge the gap between a brilliant idea and a valuable product—but only if the bridge is successfully completed and built to scale. And in the age of disruption, that doesn't always happen for many organizations with pilot projects in their portfolios.

Take internet of things (IoT), for instance. A 2017 study by McKinsey & Co. found that less than 30 percent of companies with IoT pilot projects had even started to scale. More than 80 percent of those projects were languishing in the pilot phase for more than a year. And for 28 percent of initiatives, the pilot project had dragged on for more than two years.

Of course, pilot project purgatory can strike any industry. Glacial progress, an inability to scale, or an initiative that ends without clear success measures or next steps are universal risks project professionals encounter in today's fast-paced business environment. “There are always certain challenges a project manager can face, but the challenges in a pilot project are far bigger than a normal, defined project,” says Shailendra Malik, IT program manager, Optimum Solutions, Singapore. He notes that two components that would be considered crucial to define on any other project—the schedule and the scope—are often less defined on a pilot. And without those clear parameters, it can be hard to steer the full-scale initiative toward completion.


—Shailendra Malik, Optimum Solutions, Singapore

The data supports his on-the-ground project perspective: When Innovation Leader researchers interviewed executives at companies with more than US$1 billion in revenue, they found that most pilot projects or initiatives stumble in the hand-off from the R&D team to the execution team. More than one-fourth of executives reported the transition needed serious work, while 16 percent described it as “terrible.” Why? At many companies, pilot projects might get an enthusiastic kickoff from senior management but experience little of the rigor—such as metrics, milestones and governance—needed to drive results, the researchers noted.


Pilot projects have reached a fever pitch: Companies are piloting an average of eight Industry 4.0 initiatives, such as robotics, automation, artificial intelligence and IoT. (See “Hot Spots,” page 55). It's easy to get swept up in new technologies, but will those initiatives actually impact the bottom line? According to McKinsey's 2018 Manufacturing Global Expert Survey, less than 30 percent of all Industry 4.0 pilot projects (that is, those in the data automation and data exchange space) are rolled out company-wide. Industrial automation, software and semiconductors were the sectors in which such scale was most likely to be true, with pilots in healthcare, automotive components, and paper and packaging the least likely to be widely rolled out.

“There's a natural excitement to get started with a new pilot project, but the first step should really be a business assessment,” says Guinivere Pedro, senior project manager at enterprise development consultancy Enterprise-room, Johannesburg, South Africa.


—Guinivere Pedro, Enterpriseroom, Johannesburg, South Africa


Depending on the organization and the importance of the initiative, the pilot project might have been announced with great fanfare. But even when the C-suite is driving change, project managers must secure enterprise-wide buy-in, says Kristen Risley, PMP, senior project specialist, Massachusetts General Hospital, Boston, Massachusetts, USA. Project managers should emphasize a communication strategy from the start, particularly if the pilot's success is dependent on change management.

Last year, Ms. Risley was tasked with managing a one-year pilot project. The goal: Get 26 private practices affiliated with Massachusetts General to implement an iPad questionnaire that would address patients’ social determinants of health (such as homelessness, food insecurity and lack of transportation). The pilot was part of a very visible state contract, so top-level buy-in came easily. But to ensure strong support from other stakeholders, Ms. Risley met with hospital administrators, practice nurses, receptionists and patients. She scheduled site visits, going practice by practice to understand the special needs of each facility and to serve as the friendly face of the project. Such granular outreach showed empathy for change and helped facilitate buy-in from all levels.

“You can only videoconference with so much effectiveness,” she says. “Being able to show someone exactly what's going on and explain again why we're doing this pilot is huge. Otherwise, it's not going to work.”


—Kristen Risley, PMP, Massachusetts General Hospital, Boston, Massachusetts, USA


Scale is often the biggest question of a pilot project: Does the organization have the resources to roll this out broadly? Does the technology work well enough? Is there a critical mass of stakeholders on board rather than just a small minority of early adopters? Keeping those factors in mind can help identify stumbling blocks or signal larger issues during a full-scale implementation, says Ms. Risley. But a single obstacle shouldn't prevent teams from achieving pilot project success.

When Ms. Risley was working on the iPad survey pilot project, she uncovered a need to ramp up some forms of technology. One of the 26 practices had no electronic health record system in place. “We are asking practices to either move completely over to electronic or to have hybrid paper and electronic systems,” she says. “There is fear of losing data, fear of changing workflows, fear of moving out of comfort zones. My job as the project manager is to help ease this transition, market the benefits to the change and use data to help the change.”

Then the team had to find a way to aggregate the paper data with the digital data. So her team is working on a spreadsheet system that eventually will merge with the larger iPad-collected results from all of the other practices. By making small exceptions for some teams, Ms. Risely's team was able to meet the state's March 2018 deadline of full implementation.


When a pilot project stretches on indefinitely, team members can become discouraged—something Mr. Malik recently had to combat during a 14-month pilot to develop a machine learning-based risk-assessment modeling tool at a bank for a client.

“Because machine learning models are primarily based on the data that you have, the outcome from a machine learning model may be very different from the outcome your other established standardized ways are giving you,” he says. “It could be because of the data quality from the historical data; it could be there are certain hidden assumed data sets which people know but they haven't documented anywhere. So the risk is you come out with a particular scenario that could be very different from the actual data the company expects it to be.”

That scenario threatened Mr. Malik's project, forcing the team to perform extended back testing and parallel runs with existing systems. The team might have felt as if it were running in place, but Mr. Malik felt confident that a breakthrough could be around any corner—and “it becomes the project manager's key challenge in that scenario to keep the team going,” he says. So Mr. Malik created an interactive dashboard that highlighted even the smallest of daily accomplishments with bold, colorful graphics.

“So much of machine learning and artificial intelligence projects are happening under the hood, it becomes like a black box,” he says. “The project dashboard creates this satisfaction that, yes, people are looking at the team's work, and we're making progress with this pilot.” PM

Hot Spots

Pilot project action is bubbling up around the world. These countries are taking the lead in terms of average number of digital manufacturing pilot projects per company, according to a 2018 McKinsey report:


Here's a snapshot of pilot project activity in three of those countries:


There are roughly 1,000 smart-city pilot projects ready for or under construction around the world. And China is home to about 500 of them, thanks to government priorities, according to Deloitte. However, many projects there are plagued by poor project execution, muddy strategic goals and inadequate technology implementation.


In May, the United States Department of Transportation announced 10 pilot projects that would conduct advanced drone operations—with the ultimate goal of integrating drones into the national airspace while reducing risks to public safety and security. The pilot projects could help scale drone operations for everything from package delivery to emergency management and precision agriculture to infrastructure inspections.


The capital city of New Delhi and its 29 million residents experienced just 65 “good air” days between January and June of 2018. To combat the rampant pollution, the government launched a trio of pilot projects this year, including bus-mounted filters, equipment that sucks up particulate matter and dust-separation chemical sprays. And in August, government officials said they are considering creating electric vehicle hubs in nine of the country's most polluted cities.



Related Content

  • PM Network

    Real Advantage member content open

    Artificial intelligence (AI) provides the opportunity to streamline and improve project work, according to PMI's latest Pulse of the Profession® research. Here's how top organizations and project…

  • Pulse of the Profession

    AI @ work member content open

    Our Pulse of the Profession® research reveals that artificial intelligence is already helping project leaders streamline -- and improve -- project work.

  • Pulse of the Profession

    AI Innovators member content open

    According to Pulse of the Profession® data, the impact of artificial intelligence is increasing—and it's changing how how projects are managed.

  • PM Network

    China's Net Gain member content open

    The 5G revolution has come to China, where the government views the technology as an essential way to boost its economy. From 2015 to 2018, China sponsored projects to build roughly 350,000 cell…

  • PM Network

    Innovative Outliers member content open

    By Parsi, Novid Static strategies don't last forever. Earlier this year, Uber expanded its strategy from ride-sharing to autonomous bikes and scooters. Beauty subscription service Birchbox developed and rolled out…