CONSTRUCTING LARGE SOFTWARE SYSTEMS is one of the most hazardous activities of the business world. The failure or cancellation rate of such systems is over 20 percent. Yet some large systems are finished early, meet their budgets, and have few if any quality problems. How do successful projects differ from projects that fail? Better project management and better quality control are the most important differences between success and failure in the software world. Thus excellence in software project management has a very favorable return on investment (ROI) due to cost avoidance. How does an investment into project management software factor into that ROI?
Monster Projects; Monstrous Results. Software development is a troubling technology. It is highly labor intensive and as a result large software projects are among the most expensive undertakings of the 20th century. For example, large software systems cost far more to build and take much longer to construct than the office buildings occupied by the companies that have commissioned the software. They can cost more than building a domed football stadium, a 50-story skyscraper, or a 70,000-ton cruise ship.
By “large systems” l mean those comprising 10,000 function points or more. Applications of this size are usually termed “systems” because they are far too large for individual programs. This size range is often troubled by cost and schedule overruns and by outright cancellations. Development teams of 100 or so are common, so communication and interface problems are endemic.
Software schedules in this size plateau run from three to more than five years, although the initial planning for applications of this size range tends to naively assume schedules of 18 months or less. To succeed in this size range, excellent quality control and capable project management are absolute necessities.
Even larger systems of 100,000 function points and up are among the most expensive constructs of the 20th century. This is roughly the size range of Microsoft's Windows 95 product, IBM's MVS operating system, or a major military system. software development schedules for systems of this size are usually from five to more than eight years.
“Economy of scale” doesn't apply in the case of these monster applications. While small software projects are successful in the majority of instances, the risks and hazards of cancellation or major delays rise rapidly as the overall application size goes up. Indeed, the development of large applications in excess of 10,000 function points is one of the most risky business undertakings of the modern world. (For an in-depth discussion of software project failure rates, see my book, Patterns of Software Systems Failure and Success [International Thomson Press, 1996]).
Capers Jones ([email protected]) is chief scientist for Artemis Management Systems and a leading authority on software development issues.
The Human Side of Tool Implementation: An ROI Measurement Challenge
Oh, Capers…you make it sound so easy! Yet in conversations with software vendors, IT consultants, and companies that have implemented project management tools, one phrase keeps recurring: “The human issues.” Or sometimes “the cultural issues” or “the organizational issues.” Whatever name you call it by, it all comes down to people: their attitudes, their receptivity, their skill levels. People make or break a project, it seems, whether they do it with pencils or with ERP project management modules.
Chris Vandersluis of HMS Software, for example, notes that the conversations among software development cognoscenti this year do not center, as they did in the past, around features and functions. “The truth is” he says, “95 percent of the project management software users out there don't have the skills to fully understand or make use of all the functionality that the newest software offers.” He laments the fact that many software vendors layer on the functionality—and companies purchase it—without stopping to ask, in his words: “What about deployment? Can this product's benefits actually be delivered to the end user?”
Just as the training of individuals creates an issue affecting the ROI of a tool implementation project on a micro level, the macro level is influenced by the overall maturity of the organization. Tom lsaac, of Mantix Systems, recalls fondly the days when he dealt primarily with large government contractors as clients for his program management tools: “They knew exactly what they wanted, their requirements were mandated, set, carved in stone. Now, with so many new-to-PM users, so many different application areas, they don't know what they want or need. Software has gotten cheaper, but the implementation costs are going through the roof.” Front end costs, he estimates, run at 50-60 percent of the total cost of implementation.
Even allowing for a smooth implementation process, with highly trained individuals operating in a mature project management context (we'll call this “a perfect world” for short), there's another human resource factor that can make the ROI equation a bit difficult to solve. Most vendors focus on resource allocation when trying to determine the ROI in any given client setting. Says lsaac: “If you know that to do the job presently it takes X number of people working X hours, and how much each of those hours costs you, then after implementation you can look at that same work and see, here l saved X resource-hours or X percent of resource utilization.”
But, notes ABT's Ed Farrelly, in practice that's rarely how it works: if you free up one person for a few hours, they don't disappear but may instead do the things they weren't getting around to before, or concentrate on some related, creative task that was put aside in the crunch. Tracking the migration of that brain power can be difficult; and the rewards may be difficult to measure. Still, even small gains can translate into big rewards. Farrelly cites the case of a large bank in the U.K. with about 1,500 people in the IT function. “When there's a lot at stake, if you even show a small improvement of 1-3 percent, it's a lot of money. But even more important, what you are really doing is unlocking additional time to put into the next most important project. Real value is not just cost containment, it's better resource management.”
Tom Isaac agrees: “In companies that are product-oriented, where time-to-market is very important, like pharmaceuticals, computers and other products with limited sheIf life, it's a high value proposition.”
In a perfect world, companies wouldn't try to implement processes and methodology at the same time as the software, of course, but in the current reality, any measure of ROI has to struggle with the question of whether gains or losses are due to the software…or the processes? The tools…or the people using them?
Most of those l interviewed agreed that project management software implementation could be a high-tech example of the old Hawthorne effect: If you pay attention to process and people, you get improved productivity—even if perhaps you didn't do anything else differently.
Says Ed Farrelly: “We have to realize that project management is an indirect activity: that is, the business benefits don't come directly from having the software, they come from the process of project management.”
Jeannette Cabanis ([email protected]) is acting editor-In-chief of PMI's Publishing Division.
Patterns of Project Management Tool Usage. Although software project failures outnumber successes there are successful large system built. From assessment and benchmark studies, two critical factors differentiate successful software applications from failures: (1) project management; (2) quality control.
Why do software projects run late or get cancelled? Large systems usually run late because they contain so many defects or errors that they don't work. Unfortunately, if this situation is not detected until testing begins it is too late for corrective actions to be fully successful.
Once testing starts on a defective application, the project enters a months-long nightmare cycle of finding hundreds of problems, fixing them as fast as possible, and then re-testing the application. ln fact, many troubled projects appear to be on schedule until testing begins, when it is suddenly discovered that the application just does not work.
Because software defects are the source of schedule and cost overruns, it is important to know how many defects are likely to occur. It is also important to know the most effective ways of preventing and removing software defects. This explains why successful software projects use estimating tools that can predict defect volumes.
If defects are not found until late in the project during testing, it is too late to bring the project back under control. This explains why successful software projects use formal design and code inspections prior to testing. Inspections are about twice as effective as most forms of testing in finding software defects, and they take place much earlier.
Project management and quality control are closely intertwined on successful large software projects. The secret is to keep quality under control at all times, and to use state-of-the-art predictive and tracking tools.
ROI for Project Management and Quality Control. The average cost to build a successful software application in the 10,000 function point size range is about $1,000 per function point, or $10,000,000. However, unsuccessful projects that are cancelled usually accrue costs of more than $1,500 per function point and are about 12 months late when they are cancelled. ln other words, cancelled software projects in the 10,000 function point range cost about $15,000,000 and obviously are a write-off without any positive value.
If the failing project becomes subject to breach of contract litigation, the damage costs can be much higher than the cost of the software itself.
The costs of fully equipping software project managers on 10,000 function point projects with state-of-the-art software cost and quality estimating tools is not very expensive: the cost would normally be less than $10 per function point, or only about $100,000. The total cost per manager would only be in the range of $10,000 or less.
Reader Service Number 053
The costs of performing formal design and code inspections run to about $100 per function point, or roughly $1,000,000 for a 10,000 function point application.
Thus an investment in state-of-the-art project management tools and quality control nets out to about $110 per function point. But this investment could raise the probability of a successful outcome from less than 30 percent to more than 75 percent.
Thus an investment of about $110 per function point could head off a possible loss or write-off of $1,500 per function point. This would generate an approximate return on investment of $13.64 for every dollar spent—quite a good ROI.
The presence of a suite of project management tools is not, by itself, the main differentiating factor between successful and unsuccessful software projects. Project managers who utilize a full suite of management tools are usually better trained and have a firmer grasp of the intricacies of software development than the managers who lack adequate management tools.
TOOLS BY THEMSELVES do not make successful projects, of course; capable managers and technical personnel are also needed (see sidebar). But to a very large degree, the sophistication or lack of sophistication of the project management tool suite will determine whether software projects will succeed, experience major cost and schedule overruns, or fail completely. No one in the industrialized world today would dream of starting a large engineering project of any kind without adequate tools for project management. Yet software projects whose total staffing compares to many large-scale engineering projects are routinely started using “back of the envelope” planning and estimating methods. lt is no wonder that failures are so common.
Software itself is intangible, but the schedules and cost estimates for software can be highly tangible. Software projects are still subject to the basic laws of manufacturing. Software needs to be placed on a firm engineering—and project management—basis.