Cost as independent variable (CAIV) – a CMM compatible process
Raymond L. Kile, PMP, Center for Systems Management
David C. Rolley, Center for Systems Management
The CAIV concept traces its roots to the July 19, 1995 memorandum titled “Policy on Cost-Performance Trade-offs” by the Under Secretary of Defense, Acquisition & Technology, (USD(A&T)) and subsequent command-wide practice published under the title “AFMC—Cost As an Independent Variable (CAIV).” The program management model used prior to CAIV held that cost was determined by the performance capabilities of the system (requirements), the development schedule, and the life-cycle costs associated with ownership and operation of the system. The CAIV concept shifts the emphasis of this view to one in which cost is aggressively managed and is used as one of the factors to determine the performance delivered, acceptable development schedule, and overall life-cycle costs. This view places new demands on program management and its ability to model and estimate the development and support of systems.
CMM Requirements for Planning
The Software Engineering Institute (SEI) is a federally funded research and development corporation chartered to provide “leadership in advancing the state of the practice of software engineering to improve the quality of systems that depend on software” (Paulk 1995). SEI developed the Capability Maturity Model (CMM®) to provide a method of characterizing the maturity of software development organizations. Five levels of maturity are described in the model with most organizations striving to reach Level 2, the Repeatable level, and Level 3, the Defined level. Each level has a number of Key Process Areas (KPA) that must be satisfied to be considered at that level of maturity. Further, each KPA has a number of goals and key practices that must also be satisfied before that KPA can be satisfied.
The CMM requirements for estimating are contained in two Level 2 KPAs, Project Planning (PP) and Project Tracking and Oversight (PTO) and in one Level 3 KPA, Integrated Software Management (ISM). The key practices within the CMM that are related to estimating are summarized in Exhibit 1 along with a reference to the Eight Phase Estimating Process phase that satisfies the requirements.
A Brief Look at the Eight Phase Estimating Process
The Eight Phase Estimating Process was developed in the mid-eighties in response to a need for a disciplined and repeatable estimating process that could be used for responding to Department of Defense proposals. One of the authors (Ray Kile) developed a software cost estimating model, called REVIC, which was subsequently freely distributed and used by both government and industry organizations. The Eight Phase Estimating Process (Kile 1987) was developed initially to provide structure to a training course for organizations desiring to use the REVIC model, but it soon proved far more useful as a means of elaborating the risk inherent in a cost estimate. Subsequently, the Eight Phase Estimating Process provided a number of key practices that are currently documented in the CMM®.
Phase 1. The Design Baseline Phase
The Design Baseline phase starts as soon as the System Engineers, or equivalent, have determined a candidate architecture and design for the proposed system. The requirements must have been gathered and allocated to the components within the candidate architecture. The output product from the Design Baseline Phase is a table or listing of the components along with the required functionality of each. The product from this phase should be reviewed by the appropriate level of management and placed under configuration control.
Phase 2. The Size Baseline Phase
Once the Design Baseline has been established, the next task is to develop the size estimates for the components of the system. While estimating the size, risk information can be captured in the form of either ranges in the estimate or as a standard deviation using methods like PERT. The resulting estimates should also be reviewed by management and placed under configuration control.
Phase 3. The Environment Baseline Phase
The next step is to specify the Environment Baseline. The environment referred to are the total conditions that will prevail during the development of the system being estimated. That includes both the hardware and software tools and training that will be provided by the organization, as well as the skills and experience of the personnel that will be assigned the task. Every parametric estimating model has a set of parameters that are used to adjust the model's estimates for differences in the environment. During the Environment Baseline Phase, all the information collected to date will be used along with knowledge about the organization that will develop the system to establish the appropriate settings for these parameters.
Exhibit 1. Key Estimating Practices of the CMM®
It may occur to the reader that this phase could have been accomplished at any point in time since organizations remain relatively stable in terms of their tools, training, and personnel. However the size of the product to be developed will have a definite impact on the environment for several reasons. First the organization's ability to handpick personnel and staff a project with experts and highly experienced personnel is far easier for a small project requiring only 5–10 personnel than for a large project needing 50–100 or more personnel. Similarly, the number of tools and training provided becomes diluted with size. Thus, size estimation must occur before the environment can be established.
Phase 4. The Baseline Estimate Phase
By the time we have reached the Baseline Estimate Phase, we now have all the inputs necessary to run our parametric effort/cost and duration models. Each set of input types (sizes, environments) have been independently generated, reviewed, and approved to form the associated baselines. Each baseline is linked to the products of previous phases and is traceable back to the original requirements. In this phase, we enter the input parameters into the parametric model and see for the first time the predicted effort and duration for the software portion of the project. It is at this point in the estimating process that an effort/cost or duration problem will become apparent. In the past the typical response was to somewhat arbitrarily change the size or environmental parameters to make the estimate match management's expectations. In the estimating process we now introduce a rule to preclude this break in the chain of traceability.
Rule 1: Never change the output of a given estimating process phase without a corresponding change to the inputs of that phase.
To illustrate the rule, consider the situation where we have just determined that we have a budget overrun. In order to reduce the cost and follow Rule 1, we must first change one of the inputs to the phase. In other words, we must change the environment or the size table information. We now introduce another rule.
Rule 2: Always try to change the previous phase's products first before proceeding up the process chain.
This rule says that we should back up the process chain one phase at a time when we need to make changes to the outputs of any particular phase. For the example given, where we have a cost problem, we should go to the previous phase first to try to effect a change. We dutifully revisit the environment table and see if there is anything we can change. Perhaps we decide that if we can handpick staff we can raise the productivity and reduce costs. We must then document the rationale for the change and reaccomplish the management review to establish a new Environment Baseline. When we then reenter the Baseline Estimate Phase and rerun the model with the new environmental parameter settings, we may find that the improvement in productivity now meets the cost constraint.
In accordance with Rule 1 and 2, we may have found that we had no justification for changing any of the Environmental Baseline parameters without a corresponding change to the Size Baseline, so we proceed back to the Size Baseline Phase. However, here we find that the size information in the baseline is directly traceable to the design components and their required functionality. Following Rule 1, we can't arbitrarily change the size estimates based on wishful thinking, and must go all the way back to the Design Baseline Phase. In this case, in order to reduce the cost to meet the budget, we must either eliminate a required function or down-scope the means of satisfying the requirement. In each case, we must capture the rationale for the changes and maintain a document trail for subsequent analysis.
Phase 5. The Project Estimate Phase
All parametric models come with specific assumptions about both what development life cycle phases (e.g., design, code, test) and types of activities (e.g., system engineering, project management) are included. When the characteristics of your project do not match those assumed by the model adjustments will have to be made. The purpose of the Project Estimate Phase is to add those things to the estimate that are not included in the model's assumptions and to take those things out of the estimate that the model assumes but your project will not do.
Some examples will illustrate the problem. Most parametric estimating models don't include the up-front system engineering time required to perform system level requirements analysis. If this work is to be included in the project, you must add the effort and duration required to the model's estimates. Also, many models were calibrated from government projects that had a large amount of documentation. Your project may not require that much documentation and you will have to reduce the effort (and duration). Another example of activities mismatch is the inclusion, or not, of line management in the effort predictions. Most models include the first line project manager, but do not include any other management type labor.
Phase 6. The Risk Analysis Phase
In the Risk Analysis Phase we will take all the risk information collected and try to determine in both a quantitative and qualitative manner what risks are inherent in this project associated with the estimate. This phase is usually run in parallel with the next phase, the Pricing Phase, and the risk analysis should also consider the risk inherent when management decides to price the system differently from the estimate. Note that this risk analysis is not the same as a technical risk assessment leading to a risk management plan, although the risks identified and mitigation actions planned should be included in all project plans including the project's risk management plan.
Various methods of sensitivity analysis can be performed including Monte Carlo methods, use of standard deviations to get estimate spreads, and simply varying the input parameters to give the best- and worse-case estimates. However the risk information is generated, the goal is to make the quantitative and qualitative information support managers who must trade off desire to win the contract against the possibility of an overrun in budget or schedule.
Once management has decided on the price for this system, the risk analysis takes a slightly different form. We can now go through the estimate inputs and determine what changes would be needed to produce the system for the proposed price. For example, we might say that in order to meet the proposed price we would have to have the authority to handpick all the staff. Or we might say that we need to reduce the size by eliminating or reducing some functionality. This information should then be documented in a risk memorandum and used by the project team in their risk management planning.
Phase 7. The Pricing Phase
The purpose of the Pricing Phase is to use the available estimate and risk information to arrive at an acceptable price for the project. Management has two conflicting goals during this phase. The first goal is to win the project. The second goal is to ensure there won't be an overrun of budget or schedule. As management tries to optimize probability of winning the project by lowering the price, they are simultaneously increasing the probability of an overrun. Similarly, if management wants to ensure there won't be an overrun by raising the price to include some management reserve, they are reducing the probability of winning the project.
Once management has decided on the price or bid, the risk analysis activities in Phase 6 are reengaged to determine the risk inherent in the difference between the Project Estimate and the project price. Management should carefully consider the risks and ensure that appropriate risk planning and mitigation is included in the project's plans and schedules.
Phase 8. Dynamic Data Collection Phase
The purpose of the Dynamic Data Collection Phase is to close the loop by gathering data for calibration of the model for future estimates. This includes both the gathering of data from the current project as it is progressing to help manage the project as well as adding completed project data to a database used for recalibrating the model. This phase also continuously tracks the impact on effort, cost, and duration as risks become reality.
For ongoing projects, data collection can be used to calibrate the model on the fly. Actual experience on the project through a major milestone can be used to predict the remaining effort or duration in a manner analogous to using actual cost data to calculate the Cost and Schedule Performance Indexes for Earned Value projects.
Performing CAIV Analysis Using the Process
Within the CAIV concept, the performance, schedule, and costs for a system are developed via trade studies with the participation of the user, acquisition, and development organizations. Once the value (or range of values) for these parameters has been established, management treats the cost parameter more as a constraint than a resultant. This places program management in the position of constantly evaluating the program's performance against the cost goals and understanding the impacts of changes to the program's baseline. This management style requires the development and use of system models and associated estimates to identify, manage, and mitigate opportunities and risks.
The Eight Phase Estimating Process supports this effort through the development of an evolving set of models that describe the maturing system. As described, the later phases build on the earlier phases and the required consistency from phase to phase provides visibility into the methods used by the program's management and estimators. Once a consistent set of models has been established for the program, changes in any phase can be evaluated for its impact on the CAIV goals of the program. This provides management the basis for the Project Planning capability required by the CMM at level 2.
The Dynamic Data Collection Phase provides the mechanism to update the models set as the program progresses. This mechanism supports the CMM project Tracking and Oversight capability described at level 2 of the model. It also supports the data collection and analysis capability, which starts with the establishment of the software process database under Organization Process Definition at CMM level 3 continuing to Quantitative Process Management at CMM level 4 and Defect Prevention at CMM level 5.
Summary (Main Points)
The discipline and coordination required by the CAIV concept imply the use of documented processes. As described in this paper, existing estimation methodologies can easily be used to support program management utilizing CAIV. When properly partitioned, these methodologies (i.e., the Eight Phase Estimating Process) can assist program management in identifying opportunities and risks that may relate to the assumptions about subcomponents of the development or support environment
Programs managed in accordance with the concepts of CAIV, using existing estimation techniques, practices, and tools can successfully demonstrate SEI SW-CMM compliance.
Paulk, M. 1995. The Capability Maturity Model: Guidelines for Improving the Software Process. Addison Wesley.
Kile, R. 1987. A Process View of Software Estimation. Proceedings, Revic User's Group annual meeting.
Proceedings of the Project Management Institute Annual Seminars & Symposium
November 1–10, 2001 • Nashville, Tenn., USA