Project Management Institute

Software quality with predictable numbers

by Jeet Sandhanwalia, PMP

images

THE ULTIMATE OBJECTIVE of Quality Assurance and Quality Control for a product or service provider agent/organization is to minimize the number of customer complaints for a defined set of requirements. It would be very helpful and useful if we could predict how many complaints will be generated when a product or service deliverable is provided to the customer. The provider could then focus on the processes, tools and metrics to ensure that an adequate amount of QA and QC effort is built into the project life cycle. This effort would be spent to find errors and fix/enhance the deliverable so that the numbers of defects (which would result in customer complaints) would be minimized.

What if you could accurately estimate the errors in a software product under development and accurately forecast the resources needed to fix them?

The Concept. During a project management course, 1 was introduced to a new concept of estimating the number of errors in a software product. This concept is based on the fact that there will always be errors in a software product. Thus, the project team must allocate a certain percentage of the total project effort for Quality Assurance and Quality Control in order to focus on uncovering a certain number of errors and fixing them before a product is released to the customer. The basic premise of the concept is to estimate the number of errors expected, collect metrics, observe the pattern, and document the lessons learned throughout the project life cycle.

Starting Point Assumptions. There are several practices in the software development industry to estimate the number of defects in a software product (for example, it is presumed that there are about 50–80 defects per 1,000 lines of code). The number of errors depends upon the complexity of the product and maturity of the development group developing the product, among other factors.

A performing organization can utilize any industry-recognized, sophisticated approach to determine how many defects are expected to be “inserted” into a product developed by that organization. The next step will be to determine how much effort is required to uncover and fix those defects.


Jeet Sandhanwalia, PMP, is a project manager with Electronic Data Systems in Warren, Mich. Questions on the material presented in this article can be e-mailed to him at jeet@flash.net.

Determining these efforts is a key to the whole exercise. Any organization implementing this concept has to make some starting assumptions for QA and QC Effort Percentage and the effort it takes to fix each defect during the project.

This is a plot of actual number of defects discovered and fixed during a Software Release Project #1 vs. Expected number of defects during the project duration

Exhibit 1. This is a plot of actual number of defects discovered and fixed during a Software Release Project #1 vs. Expected number of defects during the project duration.

I've utilized this concept in several software development projects. For the purpose of practical application of this concept, our project team decided to allocate 10 percent (not based on any industry standard, but more or less a gut-feel number to get started) of the total effort for QA and QC. In our application area of the software development projects, we have seen that historically it takes about three hours of effort to uncover an error and fix it. Taking these assumptions as a starting point, our project team derived the numbers, collected metrics and made observations for certain behaviors. At the end of two particular projects (explained in the Examples), we observed that results from one project were fairly accurate, whereas results from the other project were not as accurate.

Conceptual Theory. Let's assume a software development project has been authorized, with defined requirements and deliverables.

The project team has estimated the whole project for a certain number of effort hours:

1. Number of Project Effort Hours = NPEH.

Determine/estimate a starting point assumption on the percentage of QA and QC effort percentage:

2. QA & QC (Effort) Percentage = QACI?

Calculate number of total QA and QC effort hours for the project:

3. Number of QA & QC Effort Hours = NQEH = NPEH*QACI?

Many IT organizations have local standards/procedures for QA and QC. The most common method is to perform walkthroughs at various stages of the project life cycle (for example, project plan development, system design or code walkthroughs, application cross-testing). The reviewers at these QA gates or walkthroughs will probably find some errors in the deliverable, which, if not corrected, will go out in the end product, resulting in customer complaints.

Depending on the type and/or severity of the error discovered during QA gates, it may take X number of hours to correct it. Some errors will take longer to find than others:

4. Number of Hours per Error Detection and Correction = NEDC.

The theory emphasizes that the number of errors found and fixed due to the QA gate/walkthrough and QC efforts is, in fact, reducing defects in the end product and hence minimizing the customer complaints.

With the acceptance of this concept, the next step is to calculate the number of defect reductions due to QA and QC efforts throughout the project life cycle:

5. Number of Defect Reductions = NDR = NQEH/NEDC.

Depending upon the nature of industry segment, complexity of systems and customer expectations, the development team can allocate an appropriate QA and QC Effort Percentage and Number of Hours per Error Detection and Correction. This exercise will assist the project team in predicting the Number of Defect Reductions in a project product.

The major objective of QA and QC for a project is to minimize the customer complaints for a deliverable. It is probably impractical to minimize the customer complaints to “zero” in an IT project, but the following guidelines can be used for setting QACP to ensure that sufficient effort is allocated to find and fix errors, thus minimizing the customer complaints.

Quality Rating for a deliverable/product is measured in the following terms:

No. of Customer Complaints Quality Rating
0–3 Excellent
3–5 Very Good
5-10 Good
> 10 Not Good

This Quality Rating (QR) scale is limited to the User Acceptance Test (UAT) phase of the project life cycle; however, one can establish a long-term (that is, through the Operations and Support phase) QR for the product life cycle based on the criterion discussed in this article.

The project team/organization, along with customer input, makes a decision for the Quality Rating (that is, can you live with a rating of “Good” or is the goal to achieve an “Excellent” rating all the time?). Depending upon this goal, complexity of the project, and other factors such as staff experience, availability of tools, and so on, each project team can factor in the QA and QC Effort Percentage toward the overall project effort.

Practical Examples. I have used the above concept in several software development projects. The data gathered during a manufacturing system's release project is illustrated in the example below to verify the conceptual theory.

Example 1. The first example is a release project for a total effort of 435 hours and duration of 4 months. The parameters were calculated as follows:

1. Number of Project Effort Hours (NPEH) = 435 hours 2. QA & QC (Effort) Percentage (QACP) = 10% (assumed)

3. Number of QA & QC Effort Hours (NQEH) = NPEH*QACP = 435*.1 = 43.5 hours 4. Number of Hours per Error Detection and Correction (NEDC) = 3 hours (assumed)

5. Number of Defect Reductions (NDR) = NQEH/NEDC = 43.5/3 = 15.

From the basic concept and assumptions, the project team expected to uncover and correct 15 errors (in actuality, making 15 enhancements) in the product.

Expected number of errors, based on the effort hours spent each week were calculated. Similarly, actual numbers of errors discovered during the project were documented. These data points were plotted on a weekly basis, as shown in Exhibit 1.

This project was kicked off about three weeks prior to the first data point shown in the exhibit. Some effort was spent in requirements gathering and project plan development. The effort spent on these tasks estimated errors and the project team gathered actual errors discovered. However, the first data point plotted on the graph did not start until 14 July.

According to the theory, the project team should have already found four errors, whereas we found only one error, and fixed it (there was an error in the project schedule, which if not corrected, would have been presented to the customer).

The data points on the curve depicting Estimated Errors (—images— # Est. E's) were calculated based on the effort hours spent on the project on a weekly basis. As expected, this curve is fairly smooth.

The data points on the curve depicting Actual Errors (—▲— # Act. E's) were calculated based on the QA gates (walkthroughs during design and coding) and system testing phases. We found 10 errors, prior to the 6 October release date, whereas according to the theory we should have found 15 errors and fixed them prior to the software release to the customer. The product was mandated to release on this date, due to scheduling constraints at the customer site. The developers spent more effort at the customer site after implementation (prior to the production cutover) and uncovered five more errors. However, the end product still had more bugs. Overall, 12 errors were uncovered at the customer site.

According to the Quality Rating scale, this project resulted in 12 customer complaints and hence the product quality is rated as “Not Good.”

Example 2. The second example is a release project for a total effort of 770 hours and duration of 4 months. The parameters were calculated as follows:


Reader Service Number 057

This is a plot of Actual number of defects discovered and fixed during a Software Release Project #2 vs. expected number of defects during the project duration

Exhibit 2. This is a plot of Actual number of defects discovered and fixed during a Software Release Project #2 vs. expected number of defects during the project duration.

1. Number of Project Effort Hours (NPEH) = 770 hours

2. QA and QC (Effort) Percentage (QACP) = 10% (assumed)

3. Number of QA and QC Effort Hours (NQEH) = NPEH*QACP = 770*.1 = 77 hours 4. Number of Hours per Error Detection and Correction (NEDC) = 3 hours (assumed)

5. Number of Defect Reductions (NDR) = NQEH/NEDC = 77/3 = 26.

Based on the basic concept and assumptions, the project team expected to uncover and correct 26 errors in the product.

Expected number of errors, based on the effort hours spent each week were calculated. Similarly, actual numbers of errors, discovered during the project were documented. These data points were plotted on a weekly basis, as shown in Exhibit 2.

This project was also kicked off about three weeks prior to the first data point shown on the exhibit. Some effort was spent on the project in requirements gathering and project plan development. The effort spent on these tasks estimated errors and the project team gathered actual errors discovered. However, the first data point plotted on the graph did not start until 25 August.

According to the theory, the project team should have already found seven errors, whereas we found only one error, and fixed it.

The data points on the curve depicting Estimated Errors (—images— # Est. E's) were calculated based on the effort hours spent on the project on a weekly basis. As expected, this curve is fairly smooth.

The data points on the curve depicting Actual Errors (—▲— # Act. E's) were calculated based on the QA gates (walkthroughs during design and coding) and system testing phases. The period from 10-24 November shows a significant rise in error discovery rate. This was the system test phase, when the QA team performed testing. These errors were discovered and fixed, just prior to the 1 December release.

According to the theory, we should have found 26 errors during the project cycle. But, during the implementation week, the customer found five errors in the system. This means the project team should have found at least two more errors and fixed them in the product in order to achieve an “Excellent” quality rating. But, in fairness according to the Quality Rating scale, the product quality is still “Very Good.”

THE CONCEPT AND PRACTICE for gathering quality metrics discussed here are relatively easy to understand and follow. Any organization, implementing this concept has to make some starting assumptions for QA and QC Effort Percentage and the effort it takes to fix errors during the project. It may vary from project to project (for example, it may take six hours for a new person to fix an error as compared to three hours for a veteran). Similarly the complexity of the project may dictate different assumptions.

Conclusions from this type of data depend upon each organizational and project characteristics. The project manager may want to observe certain behaviors of the data (for example, if the number of actual errors is too high, it may be possible that the resource(s) working on the project are not adequately familiar with the technology, tools, or not trained enough to handle this kind of project). At the same time, if the project team is not finding any errors, it may be an indication that the quality review process may not be adequate or the project is just too simplistic. Again, these types of conclusions will be highly dependent on the organizational and project characteristics.

The data gathering in the above two examples was stopped after the User Acceptance Test conclusion at the customer site. One can continue to gather the data and follow up on the errors uncovered during the operations phase to determine the overall product quality throughout the life cycle of the product/release.

The performing organization needs to develop a historical database with data gathered for a number of projects over a period of time. This data can be analyzed to adjust and massage the original assumptions (that is, quality effort percentage and hours required for error detection and corrections, and so forth.) and fine-tune these parameters. Although I've discussed the IT industry environment here, the same concept can be applied to any industry where a set of deliverables is due to the customer. images

This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI.

PM Network July 1999

Advertisement

Advertisement

Related Content

Advertisement