Software project audits
a launching pad
This article describes approaches to conducting internal or external audits of software development projects. Audits may be conducted to help in recovery planning for a failing software project, or on a regular basis as part of a failure prevention approach. Guidance for scheduling audits on a preventive basis is presented. Recovery planning approaches are described.
An Ominous Sense of Disaster
You're developing a new software program and your project team or contract developer has slipped the schedule. Should you be concerned? The answer is, probably not if this is the first slip. Software projects are more difficult to manage than construction projects, and would you expect to completely go crazy if your home remodeling project slipped a few weeks. If half of your software projects are coming in on time and on budget or better, you're doing pretty well.
You should be more concerned after the second or third slip. This is a definite sign of a project potentially in trouble. You might think that the trouble is the schedule delays, but this is merely the surface manifestation of a deeper underlying problem. Schedule slips and cost overruns may increase your costs and delay delivery, but in most cases they will not result in total project disaster. Unfortunately, when you look beneath the surface of a project missing deadlines you will often find that the underlying architecture and code itself is seriously, perhaps even fatally, flawed. There are two possible reasons for this correlation:
- The most frequent explanation is that the developers are over their heads. They are attempting to build a system whose complexity exceeds their experience or ability (or both) and the result is a flawed architecture, incorrect object design, poor database design, inefficient code or data access, and so on. Don't get me wrong, these individuals may have the best of intentions and be competent in development in general, but for this particular complexity of application, they are lost in the woods.
- A much less frequent but not uncommon explanation is that the developers have the capability to build the system, but the initial estimate of effort and time was so badly under scoped that they do not have anywhere near enough time to do the job right. Managers may exert so much pressure on the developers that they crumble and produce poor quality architectures, designs, and so on simply trying to meet unrealistic milestones. This is one of the reasons development shops should emphasize to their programmers that they must never sacrifice quality for schedule, and that it is their jobs as professionals to stand up to any management pressure otherwise. Schedule slips with high quality code are always preferred over on-time performance with poor quality (not that anyone likes schedule slips).
If you ignore the initial warning sign of multiple schedule slips, then you have laid a foundation for total project failure and cancellation. This will show up when the system is finally delivered in one or more of the following forms:
- The system has numerous defects and crashes or operates incorrectly to the extent where it is not usable;
- The system is missing key functionality that is necessary for it to be deployed operationally;
- Seemingly minor enhancements to the system are very difficult and costly to implement and often result in unexpected problems in other parts of the application; or
- The system performance is sufficiently slow that it is not feasible to deploy it operationally.
If you suspect that you may have a project going down this path, you're not alone. According to a Standish Group Study 40% of software projects underway now are expected to fail, and 33% of all projects are over budget and late.
Project Auditing Methodology
Suppose you suspected that one of your subsidiary corporations was in trouble financially, and that they were intentionally or unintentionally hiding the magnitude of the problem within their accounting department. Without hesitation you would call in outside experts (certified public accountants) to do an audit and tell you where you really stood.
Similarly, if you suspect that one of your projects is in trouble, you need to immediately call in outside experts to do an audit and tell you where you really stand. The audit team should be composed of very, senior managers and software engineers. Audits typically last between 3 days and 3 weeks, based on the size of the project.
The audit consists of a management audit and a technical audit. Typically, on smaller projects the technical audit is the most critical and the focus of the audit, while on the larger projects (over $5M USD) the management audit dominates.
The Technical Audit
The technical audit focuses on the design team, and to a lesser extent, the programmers doing the actual implementation. It begins by looking at the overall system architecture and database design. The question is not really whether these are right or wrong, but rather whether they are appropriate to the nature of the application (usage, transaction volume, database size, planned evolution, and so on). Our experience has been that if these two elements are correct, the project has a solid foundation and if there are other problems, salvage is possible. On the other hand, if these two elements are incorrect then the remainder of the system is likely to need a total rewrite. Running a close third in importance is the design of the objects and business application servers. If these are wrong, the system can often be made to work but maintenance will be difficult. A decision will need to be made whether to fix and deploy the current system while immediately redesigning a follow-on system, or to redesign at this point in time.
Once the design has been reviewed, the implementation must be examined. The process begins by using automated tools to look at comment density (both in headers and embedded within functions) across the application as a whole and by function or module. A similar analysis of McCabes complexity metric for each function is completed. Functions with high complexity are candidates for simplification and are likely trouble points for defects. The code itself (including data access code such as SQL statements and stored procedures) is then examined, either in its entirety or on a sampling basis. The audit team looks for inefficient coding techniques, proper error and exception handling, duplicate code blocks (duplicate code should be encapsulated in a function or object, not just cut and pasted), and other obvious problems. Finally, the user interface is examined for usability and conformance with industry standards. Of all items mentioned, the user interface is the easiest to fix if deficient.
The Management Audit
The management audit has three steps. Gather metric oriented input date; prepare project baselines using industry standard approaches; and compare actual or projected values with the resultant baseline. The project baseline will include things like total effort and schedule, deliverables (including page counts), labor loading curves over time by skill set, development team skills and experience, maintenance projections, defect projections by category, and so on. By comparing historic values for staffing and other metric values with baseline values, deviations can be identified and analyzed. Similarly, forward looking project plans can be compared to the baseline values and deviations examined. Examples of the types of problems that pop out are shown in the following table:
|Metric||Industry Std.||Audit results|
|Software Design Description page count||2,110||493|
|Software Test Description page count||873||42|
|Software testers (person months)||72||14|
|Software integration and test time (calendar months)||4.2||0.75 (planned)|
Sample management audit problem areas
Disaster Prevention Audits
Through periodic audits throughout the project lifecycle, problems can be identified early and corrective action taken in a timely fashion. These preventative audits have significantly increased the project success rates on large projects in the State of California, and are now a requirement for all large projects. At each audit the team will look at work performed to date, and plans for future work, to identify problems (if any) with each area. Preventative audits are normally accomplished as follows:
|Milestone||Scope of Audit|
|Project Initiation||Project baseline plans|
|Software Requirements Review||Requirements, architecture, plans|
|Software Design Review||Scope creep, design, architecture, database, interfaces, test documentation and approach, coding guidelines, plans|
|Completion of coding||Code implementation (complexity, adherence to guidelines, encapsulation, algorithm order of magnitude, plans, user interface|
|Delivery||Maintainability, conformance to requirements, usability|
In addition, project audits are normally conducted as a minimum every 6 months so if one stage extends longer than 6 months a progress audit would be conducted during the middle of the phase.
Three Actual Examples
Let me describe three actual project audit results by way of illustration. The names are hidden because of nondisclosure agreements.
- Project Alpha was a large project (over $100M). We focused on the management audit and identified inadequate staffing levels overall, incorrectly trained staff, and incorrect labor mixes. Our technical audit indicated abnormally high defect levels and major units needing rewrite. The project was subsequently cancelled.
- Project Beta was a large project (over $150M). Our technical and management audit indicated that the schedule slips were reasonable, the quality was high, and the team was doing a good job of recovering from an initial underestimate of the scope. We also identified that the planned maintenance effort was significantly underestimated. The project continued with a new schedule, was successful, and an updated maintenance budget was submitted.
- Project Charlie was a small to moderate project (approximately $1.5M). Our technical and mangement audit indicated that the current project team and approach was failing, but we developed a recovery plan, which was subsequently implemented. Recovery planning is addressed in the next section.
Saving Your Job and Sanity – Recovery Planning
OK, suppose that the project audit determines that your project is indeed in a disaster situation. What are your options? If you continue on your present course, our experience indicates that in 100% of the cases the project will ultimately fail. Alternatively, you can cancel the project immediately and cut your losses. For non-mission critical systems that do not have a large return (in terms of reducing costs or increasing revenues) this is often the best option. In many cases, however, failure is not an option. In those cases, recovery planning is the next phase.
Recovery planning begins with software triage. Triage is a military term used by medical personnel following a major battle. The wounded are separated into three categories: Those that will get better on their own; those that will die no matter what is done; and those where medical attention has a significant probability of helping the person to live. The doctors then focus all of their attention on those where their assistance will make a difference. The initial step in recovery planning is to conduct a triage on the existing software project. What is usable as is? What can be economically fixed and used? What should be discarded? During this step, the core system functionality is also identified and all extraneous features that can be deleted or delayed are called out.
The recovery planning phase then involves development of a new, zero based project baseline plan. Existing plans, schedules, and so on are thrown out and the project is planned from the current situation to an achievable completion. This requires that formal techniques be used for estimating effort, schedule, the time-cost trade off, and so on. Delivered functionality and other project estimating parameters are adjusted until an acceptable completion date is achieved or it becomes obvious that no acceptable completion date is possible. We use Cost Xpert to complete the project estimates. We've never had a project that was formally estimated and planned with Cost Xpert subsequently fail.
Outside project audits are critical for any project that is suspected of being in trouble, and are a high return on investment (ROI) investment as a risk reduction technique for all projects of any significant size. Auditors should be independent of the project team and have no vested interest in the project succeeding or failing. Auditors should not be outside development shops, who may have a vested interest in disparaging the current project team to place their own staff on the project. Audits look at both technical and management issues, identify potential problems, and make recommendations. If a project is found to be in a failure state, then recovery planning is undertaken to try to salvage the project, if possible.
© 2004, William Roetzheim
Originally published as a part of 2004 PMI Global Congress Proceedings – Buenos Aires, Argentina
Hurricane Katrina decimated thousands of buildings in New Orleans, Louisiana, USA, in 2005, including a U.S. Department of Veterans Affairs (VA) medical facility that served approximately 40,000…
Federal Project and Program Management Community of Practice (FedPM CoP) – How Sharing Best Practices Can Lead to Success
Recognizing the value of a community focused on project practice capability and how such a community could help improve the performance of departments across the U.S. federal government, the leaders…
Developing a Project Management Office in the Department of Energy, Energy Information Administration
This case example, a supplement to the report, PMIAA: Strengthening the Government Delivery Foundation, highlights project and program management capability building within The U.S. Energy…
Commissioned and supported with research from PMI, MIT’s Consortium for Engineering Program Management, and others, this report distills how many government agencies have been leading (and continue…