Introduction
Quality Management is an important element of an organization's overall project management system. A Guide to the Project Management Body of Knowledge (PMBOK® Guide) defines Project Quality Management as the processes required to ensure that the project will satisfy the needs for which it was undertaken (PMI, 2000). Quality Assurance includes both the organization focus (continuous process improvement) and the project focus (management of the project and the products and services that are produced). Quality Assurance continues throughout the full project life cycle, starting when the project is first proposed, and continuing through project execution and closeout.
This paper describes Quality Assurance during the project execution, or delivery, phase. It describes quality assurance processes and techniques for managing a large program of related projects. To illustrate the techniques, we use examples from a $30 million-dollar software development program that consists of approximately 20 projects over multiple years.
Why is Quality Management Important?
Large software development programs are complex and inherently risky. Projects will be at various stages of completion, managed by different project managers, with diverse project teams, stakeholders, environments, and business problems. Each project is unique. Therefore, projects must use common project controls and processes to mitigate risk and to avoid repeating the same mistakes.
Exhibit 1 demonstrates the value of Quality Management for an actual software development program. The chart shows the health of the portfolio over a two-year period. When we first introduced a rigorous quality assurance process, approximately 80% of the projects were healthy versus 20% troubled. In October 2001, the portfolio reached 100% healthy for the first time ever and remains that way today.
• How did we improve quality for this portfolio?
• Introduced earlier and more effective quality assurance, with strong senior management support
• Improved scope baselines and estimates
• Improved risk management and the issue management process with the customer
• Introduced a “phased functionality” approach to software development.
The remainder of this paper describes some proven techniques that you can use to improve the quality of your own projects. The techniques work best for related projects under one program or any set of projects that are managed in a coordinated way.
Quality Management During the Delivery Phase
Quality Assurance (QA) during project delivery is a series of evaluations by seasoned, experienced project managers. In IBM, the QA professionals are certified by both PMI® and by the IBM accreditation program. These individuals should be independent from the project team to provide the necessary management and customer insight into the project. Their objectives are to assess the status of the project, assist the Project Manager (PM) with problem identification, and recommend possible solutions.
QA plays an important role by identifying potential project issues before they cause problems thereby helping to keep projects on time and on budget. At IBM, internal evaluations occur at various designated checkpoints throughout the project life cycle. There are three types of evaluations during the project delivery phase:
• Initial evaluation—Does the project have the right plan, resources, and supporting detail to execute successfully?
• Ongoing evaluations—How is the project tracking against the plan, contract, and customer expectations?
• Deliverable evaluations—Do the deliverables meet the contractual obligations?
Initial Evaluation
The initial evaluation provides an assessment of the project to ensure it gets off to a solid start. Depending on the size of the project, the initial evaluation can take anywhere from a couple of hours for a small project to several days for a large multimillion-dollar contract. Two techniques are very effective during the initial evaluation:
• Verify the basic project management processes are in place to ensure future success
• Develop and train the project team to be successful.
Project Management 101
Basic project management processes and procedures are always critical to success. QA should verify these are in place during the initial evaluation. Some of the important areas to review for software development projects are:
• There is an approved contract and Statement of Work (SOW). These documents are essential for managing the customer expectations and customer dependencies.
• The scope baseline is firm. A change management plan and change control process are in place.
• There is an approved financial baseline, including cost contingencies.
• There is a detailed project plan (schedule) that identifies all dependencies and milestones.
• The resource plan is complete. Are specialized resources (e.g., architects and designers), contractors, and customer resources identified and assigned to the project?
• A risk management plan is in place. What is the process to monitor and evaluate risks?
• There is a communications plan that includes the project team, the customer, and the project stakeholders.
• Appropriate performance reporting, issue management, and escalation processes are in place.
• The testing strategy is well defined. Are testing experts involved early in the project?
Exhibit 1. Portfolio Health Example
It is critical that proper project controls exist for communications, change control, performance reporting, issue management, and risk management. To the extent possible, controls should be common across all projects in the program. Successful project managers will reuse their processes and procedures; these PMs can “lead the way” to help establish the best practices for your program.
Building the Project Management Discipline
World-class companies use a variety of training and development programs to increase quality (Hodgetts, 1998). The emphasis is to help team members increase their potential and develop high performance, productive teams. One of the themes of this paper is to build your organizational capability and project management discipline by providing leadership through the quality assurance process.
The initial evaluation is an excellent place to start. Building capability during evaluations is a valuable technique that is often overlooked. Many organizations focus on the “audit” aspect of QA. In fact, QA professionals are often in the best position to provide coaching and mentoring for the project team, as well as share their experiences. The key is to establish the proper working relationship up front. Frame offers an excellent perspective from his own experience (Frame, 1994, p. 264):
“Successful evaluation hinges on creating a nonthreatening environment so that people are willing—perhaps even eager—to share with others the details of some of the problems they are encountering in their work.”
The initial evaluation is the right time to create a positive environment for future QA activities.
Ongoing Evaluations
Once the project has started, ongoing evaluations provide an independent assessment of the project status. Ongoing evaluations should be performed at regularly scheduled intervals and major milestones. The purpose is to verify that the project is tracking against the plan, being managed in accordance with the contract, and satisfying the customer's expectations. Techniques to use during the ongoing evaluations include:
• Hold monthly delivery reviews with the project team
• Track progress using earned value analysis
• Implement an effective risk management plan
• Divide the project into phases and manage by major milestones.
Monthly Delivery Reviews
Conducting a monthly delivery review with the project team is a basic technique to review how things are going and to measure the pulse of a project. Many organizations use a checklist to confirm the team follows the project management processes. Regular delivery reviews also give an early warning when the project is not going well, and allows sufficient time to recover before the project becomes troubled. Here is where an experienced QA professional who has built a relationship with the team can provide tremendous value by advising how to deal with issues and how to fix problems.
Exhibit 2. Earned Value Example
Software development projects have unique characteristics and risks. The delivery review should include a discussion with the project team when any of these apply to your project:
• Changing scope baseline—Information Technology (IT) projects have a propensity for change. Warning signs to watch for include: diverse stakeholder groups, unrealistic customer expectations, or inadequate participation by business areas.
• Resource availability—Any of the following conditions may cause resource issues: a shortage of skilled resources, specialized resources multitasking on several projects, a high proportion of remote or contractor resources.
• Technical challenges—There are usually technical risks associated with IT infrastructure, integration, and new technology. Watch out for “bleeding edge” technology and “version 1.0” of anything.
• Development methodology—Rapid Application Development projects must be carefully planned and controlled. Recognize and avoid the pitfalls.
• Multiple vendors—There are more lines of communication to manage as the number of business partners, vendors, and business units increase.
• Customer readiness—The customer may not be prepared to take on a new IT system once the project is complete. This is especially true when the project requires business process reengineering or organizational change.
Make delivery reviews and the associated checklists an integral part of the quality management plan for your program.
Earned Value Analysis
Project managers who use earned value analysis generally have better financial control over their projects. Earned value integrates a project's scope, schedule, and financial baselines and gives an objective assessment of the project performance. Earned value allows the PM to accurately track the work completed, identify variances from the plan, and take corrective action as necessary.
Tracking the earned value critical indicators is a useful technique to spot trouble early and respond quickly. The Cost Performance Index (CPI) measures the “burn rate” at which a project is spending money (Frame, 1994, p. 253). CPI is a critical index. Many studies by government and NATO organizations argue that once a project passes the 10% complete point, it becomes difficult to improve a CPI of less than 1.0. We also know a project's CPI is quite stable once the project passes 15% complete (Hatfield, 2001). Therefore, starting at the 15% complete point, calculating the Estimate at Completion (EAC) using the formula EAC = BAC/CPI gives a good forecast of the project's final cost.
Exhibit 2 shows the earned value data for an application development project. The original Budget at Completion (BAC) was $733k. The earned value analysis in April 2001 reveals the project is overspent and behind schedule.
The project in Exhibit 2 had a CPI of 0.8 in April 2001. Thus,
EAC = BAC / CPI
= $733k / 0.8
= $916k
Fortunately, the earned value analysis revealed this variance early. The customer eventually agreed to pay for the full overrun through the change control process (the final cost of the project was $929k). It is unlikely the problem would have been detected in time to recover had the project not been using earned value.
Put in place earned value analysis starting on day one, track progress through critical indicators such as CPI, and quickly move to correct any variances.
Risk Management: Identify, Evaluate, Mitigate, Monitor
Software development projects tend to be high risk, yet some project managers do not view risk management as a critical activity. The converse is actually true: risk management can be an extremely valuable technique when performed in a proactive, disciplined manner. Risk management can uncover opportunities to improve a project's schedule, financial results, and even scope.
Encourage your project managers to use a proactive risk management approach:
• Perform the risk identification, risk evaluation, and risk mitigation steps before the project starts (during the SOW development is an ideal time).
• Ensure the project team maintains an up-to-date risk management plan and monitors risk on a regular basis.
• Discuss the project risks at the monthly delivery review.
• Consider including high-impact risks on the project status reports to keep these risks visible to the project stakeholders.
• Implement the planned action when a risk event occurs. Some risks may become issues—when this happens remember to raise and track the issue using the issue management process.
Troubled projects deserve special consideration. When a project is in trouble it is a good idea to repeat the risk management process outlined in the PMBOK® Guide. You will usually find new risk events and greater risk impacts, leading to an overall increase in risk for the project. In cases where the risk becomes extreme (e.g., financial loss), this may trigger a review of the business decision whether to proceed with the project.
Manage by Milestones
Software development can follow many different life-cycle models, for example, the traditional waterfall model, prototyping, or incremental development (Vliet, 1994). Each model contains a series of phases. A project following the waterfall model might include phases for requirements analysis, design, implementation, and testing. Another example of phases is a set of application releases or “builds” produced by incremental development.
The PM could develop a complete project plan and estimates for the entire life cycle. A different technique is to divide the project into a series of phases corresponding to major milestones. Each phase is three to four months long and each phase essentially represents a small project. Now the PM can manage the project by major milestones. There are significant benefits to this approach:
• There is a high degree of confidence in the schedule, cost, and scope baselines at the beginning of each phase.
• The end of each phase is a natural checkpoint to confirm the scope and estimates for the next phase.
• Estimates will be more accurate as more data becomes available after each phase.
• Customer expectations can be managed more effectively.
• Related projects can interlock at major milestones.
There are two ways to structure your project using a phased approach:
1. Divide the project into phases. Develop a milestone chart that shows the start and completion dates of each phase, along with the major milestones and dependencies. Now treat each phase as a separate project with its own SOW. The project plan includes an activity to build the SOW for the next phase.
2. Use rolling-wave planning to refine the estimates at the end of each phase. Whitten suggests that projects should not make long-term commitments beyond about three months (Whitten, 2002). Instead, the project plan includes activities to resize the project. After each phase, the project team revalidates, reestimates, and rebaselines the project plan. Resizing helps to make problems visible so they can be solved effectively and reduces the damage caused by missed dates (Whitten, 2002).
The application development project in Exhibit 2 was suffering from scope creep. The project team worked with the customer to prioritize the requirements, and then divided the project into three releases (phases). High-priority functions needed for year-end business functions went into the first release. Low-priority functions followed later in the third release. The customer paid for all the additional functionality and the team received very favorable comments for the project sponsor at the end of the project. This is an example a “phased functionality” approach to software development.
Deliverable Evaluations
The deliverable evaluation is a final QA checkpoint at major milestones and before project closeout. This evaluation provides an objective assessment to verify the major deliverables satisfy the customer's requirements as defined by the contract. Techniques used during the deliverable evaluation include:
• Conduct a readiness review for major deliverables
• Document and share the lessons learned.
Readiness Review
It is sometimes valuable to perform a detailed readiness review of a deliverable. Consider this technique when:
• The deliverable is large or complex.
• There is a major hand-off point, for example, to another project team at a different location.
• An IT system is ready to implement into production.
The readiness review is a quality gate. It occurs before the team submits the deliverable into the acceptance process. The purpose of the readiness review is to confirm the deliverable meets the customer's requirements and the SOW specifications. Here are some guidelines for the readiness review:
1. An independent, senior resource who is experienced with the technology should conduct the review.
2. All action items from the review must be documented and resolved before the deliverable is considered complete.
3. If required, verify that adequate plans are in place to measure the quality of the deliverable.
4. Readiness reviews provide excellent education and mentoring opportunities for junior team members. Take advantage of the review to build the capability of the project team.
Like the monthly delivery reviews, readiness reviews are an important part of the program's quality management plan.
Lessons Learned
A learn review should be held at the end of all large projects. Learn reviews are an important technique to avoid repeating mistakes on the same program or with similar types of projects, and to understand root causes of troubled projects. The participants include members of the project team and may include the customer. Capturing the lessons learned is important final step for the project team:
• To improve what the team delivers for their next project
• To understand what worked well and what did not
• To improve the best practices across the program.
Encourage your project managers to schedule a learn review and publish the results as part of their project closeout activities.
Conclusion
This paper described proven techniques that you can use to improve the quality of your own projects. It is important to emphasize the personal side of Quality Management along with the basic project management processes.
Throughout the Quality Management process and all the evaluations, the Quality Assurance team uses their experience to advise the project team and to make recommendations to address risks and issues. The goal is to provide advice and counsel to the project. Large multiyear programs offer excellent mentoring opportunities since many of the project team members will likely move on to new projects within the same program. The ultimate success is creating a positive environment where project teams are eager to share their best practices and the details of their work.
