Estimating with the CMMI
The Capability Maturity Model Integration (CMMI®) expands best practices from the software-only realm to include system engineering and other disciplines. It has significant requirements for estimating and tracking project's effort, cost, and other parameters. Many of the specific practices in the project planning and project monitoring and control process areas were derived from a paper (Kile, 1986) published in the mid 1980s by the author. The paper described an eight-phase estimating process that was solely intended for software estimating. The concepts have evolved over the years to include estimating systems consisting of hardware, software, and services. This paper will cover the CMMI requirements for estimating and show how the estimating process has evolved to include hardware and systems. A unique demonstration of the concepts will be presented using the public-domain REVIC software cost model. A threaded scenario will be presented and the audience will have a chance to influence the demonstration and results. The demonstration will show use of a cost model during feasibility, proposal, and ongoing project phases. A small version of this interactive demonstration has been used in estimating training for over 15 years and never fails to amaze people with how cost models can help project managers throughout the project.
The Capability Maturity Model Integrated
The CMMI® developed by the Software Engineering Institute (SEI, 2002) is an outgrowth of the work sponsored by the U.S. Department of Defense to create a method for determining the relative risk of awarding contracts to development organizations. The initial Capability Maturity Model (CMM®) addressed primarily software development organizations and only covered the activities associated with software engineering and development. It was mostly silent on the means by which original system requirements were developed and subsequently allocated to hardware and software components within the system. In addition, it did not address the issues associated with systems engineering and integration of hardware/software components. However, it provided guidance on how to achieve process-based maturity and greatly improved the efficiency and productivity of organizations that achieved high levels of maturity under its rating scale.
After the initial publication of the CMM®, many additional capability maturity models were developed for other system development disciplines such as systems engineering, software engineering, software acquisition, and integrated product development teams. These models were useful in helping organizations improve and meet their business goals, however the differences in architecture and approach between these models proved problematic to organizations striving to improve across a number of disciplines. The solution was a set of integrated models as currently reflected in the CMMI®. The new model covers the additional disciplines of system engineering, software engineering, software acquisition, and integrated product and process development. Thus the new model addresses organizations that span the spectrum of software intensive systems development and can be applied to both hardware and software development as well as maintenance and service type organizations.
Structure of the CMMI®
The CMMI®is structured with a number of process areas (PAs) that group related activities associated with a specific activity such as planning or controlling. These PAs are further structured with a number of Specific and Generic Goals which are required to be satisfied in order to say an organization is meeting the intent of the model. Goals contain a number of practices or activities that are expected in order to say the goal is being satisfied. The combination of Specific and Generic Goals that are satisfied determine the resulting Capability Level or Maturity Level of the organization.
Estimating Related Components of the CMMI®
While there are a number of estimating related components throughout the integrated model, the primary ones of interest for this paper are those associated with the Project Planning (PP) and Project Monitoring and Control (PMC) process areas.
The purpose of the Project Planning process area is to establish and maintain plans that define project activities. Within the Project Planning process area there are three Specific Goals (SG) with associated specific practices (SP) as shown below. The SPs shown in Bold are those of particular interest to estimating. Exhibit 1 contains a further amplification of these practices.
|SG 1 Establish Estimates|
|SP 1.1 |
|Estimate the Scope of the Project |
Establish Estimates of Work Product and Task Attributes
Define Project Life Cycle
Determine Estimates of Effort and Cost
|SG 2 Develop a Project Plan|
|SP 2.1 |
|Establish the Budget and Schedule |
Identify Project Risks
Plan for Data Management
Plan for Project Resources
Plan for Needed Knowledge and Skills
Plan Stakeholder Involvement
Establish the Project Plan
|SG 3 Obtain Commitment to the Plan|
|SP 3.1 |
|Review Plans that Affect the Project |
Reconcile Work and Resource Levels
Obtain Plan Commitment
The purpose of Project Monitoring and Control is to provide an understanding of the project's progress so that appropriate corrective actions can be taken when the project's performance deviates significantly from the plan. Within the Project Monitoring and Control process area there are two Specific Goals (SG) with associated specific practices (SP) as shown below. Again, the SPs shown in Bold are those of particular interest to estimating and exhibit 1 contains a further amplification.
|SG 1 Monitor Project Against Plan|
|SP 1.1 |
|Monitor Project Planning Parameters |
Monitor Project Risks
Monitor Data Management
Monitor Stakeholder Involvement
Conduct Progress Reviews
Conduct Milestone Reviews
|SG 2 Manage Corrective Action to Closure|
|SP 2.1 |
|Analyze Issues |
Take Corrective Action
Manage Corrective Action
The Eight Phase Estimating Process (EP2)
All estimating is based on predicting future performance based on past experience. New projects that are similar to past projects are relatively easy to estimate. The problem becomes how to estimate projects that are dissimilar to anything we've done in the past and the answer lies in a logical and repeatable process that systematically decomposes the problem into components that are similar to things we've done in the past and isolating the new things. The similar things are then estimated easily using analogy-based methods while the new things are estimated using a combination of parametric estimating based on attributes and informed guessing. The discipline of the processes used to perform these steps is key to credibility and repeatability of the resulting estimates. The original eight phase estimating process was developed specifically for software systems. It was based on a logical approach to isolating the various inputs to the estimate to allow a focus on their accuracy. It particularly stressed the need for appropriate levels of management review throughout the process in order to maintain the cause-effect relationship between the requirements-size-environment-constraints-estimate chain. The process made it particularly easy to handle various types of constraints in a logical manner.
The following high-level description of the eight phases illustrates how the concepts can be expanded to include systems composed of hardware, software, and service components.
Phase 1 - The Design Baseline
The cornerstone of good estimating is an understanding of the scope of the work being contemplated. In the original EP2description a table identifying the software configuration items (SWCI) was described. Each SWCI could be decomposed to the level necessary to identify components that might be similar to things built previously or were unprecedented. The table could identify the previous components as a reference for use in later estimating by analogy and prioritize the unprecedented components by risk. In moving to a Systems process it is only necessary to add hardware and service type configuration items to the table. A management review of the design baseline ensures that all stakeholders are agreed to the scope of the system to be estimated and prevents misunderstandings later.
Phase 2 - The Size Baseline
In the size baseline phase the task was to go through the design baseline table and assign a size to each software component. Size information could be derived using any of the accepted methods from analogy to parametric methods and was generally expressed as either lines of code or function points. Risk information was expressed in size uncertainty with either a range or a standard deviation. Expanding the concept to include hardware and services requires expanding our definition of size attributes to include those typically in use for hardware estimating. Various hardware parametric models (e.g. SEER-H) have been developed which use volume, density, power, weight, and numerous other attributes as key drivers for estimating hardware effort and schedules. Adding these types of components to the size baseline phase requires adding the appropriate fields in the table for hardware attributes and then estimating their values. As in the previous phase, an appropriate level of management review ensures that all stakeholders are agreed that the size expectations for the systems have been captured appropriately.
Phase 3 – The Environment Baseline
Once the scope and size information has been determined, the next step was to specify the development environment that would be used in the project. The environment phase follows the size phase because size in a major driver of the environment available to develop the product. It might be reasonable to have the company expert do the development of a small software component on the newest workstation with the latest software tools, but will that same environment prevail if the team size is increased to a hundred or more developers? In most organizations the answer is no and it is clear that the environment is not a constant in any organization. The environment should be specified in terms that are consistent with the parametric cost model being used. In the software world we are used to talking about personnel experience and capabilities, product constraints, and a host of other cost drivers. These cost drivers are represented in the estimating models by parameters that can be set to the desired value for the individual cost driver. The original COCOMO model (Boehm, 1981) had fifteen parameters and described many others. The Revic model added a few more and other newer models have added many more. Some of these cost drivers are found in the hardware estimating models, but different ones are also present. Expanding the process to include hardware and services requires adding these additional cost drivers to the table to be specified. Again, an appropriate management review ensures stakeholders agree the development environment has been specified properly.
Phase 4 – The Baseline Estimate
Once the size and environment information has been reviewed and accepted, the next step in the process is to input the information into the software model being used and document the results. Expanding the process to include hardware and services would require running multiple models to estimate the effort and duration of the hardware and service components separately from the software. Typically, the management review following this activity is the first time that problems with effort, cost, or schedule are visible. In undisciplined organizations it was common for estimators to change the input parameters without any real justification in order to make the estimate agreeable to management. The EP2 was designed to ensure that management is part of the process with the requirement for a review after each phase. By the time management realizes that the estimated effort or schedule is unacceptable, they have already agreed that all the inputs are accurate to the best of the team's ability to specify them. Since we have taken pains to ensure the chain of cause and effect from the design through environment, if the resulting effort or schedule is unacceptable for any reason the only recourse is to re-visit one of the previous phases and try to make the situation better. The first step would be to try to improve the environment by hand picking personnel, improving tools, or reducing constraints. In order not to do this ad-hoc, the EP2 requires documenting both the change and the rationale to maintain an audit trail. If that's not possible, the next step is to try to reduce size. However it usually requires a design change to effect a reduction in size. Typically this is what usually happens and we see it in reduced scope, delayed deliveries, or subsequent builds being planned. At the management review all stakeholders will agree that the effort and schedule were derived logically from the input size and environment and responsive to any constraints that were applied. We have also built an audit trail showing how the estimate has been changing over time.
Phase 5 – The Project Estimate
Since all parametric cost models are only approximations of real life, their assumptions about the development need to be compared to the actual project and the estimated effort/schedule revised accordingly. The models assume a development lifecycle with a given start and finish. If the actual project starts and ends differently from the assumptions in the model, the estimates will need to be revised. Similarly each model makes assumptions about the types of labor activities that are included in the estimates. For example, some models don't include management while others exclude system-engineering effort. The model's estimates must be adjusted to match the actual project by adding or subtracting effort and schedule as appropriate to reconcile the assumptions to the reality. Only at the conclusion of this phase do we have a complete estimate for the development project's complete effort and schedule. Expanding the EP2 to include hardware and systems only exacerbates the problem since there are now additional model's whose assumptions must be understood and reconciled to the proposed project. The management review should pay particular attention to the additions and subtractions necessary to reconcile the model's assumptions with the project being estimated.
Phase 6 – The Risk Assessment
This phase and the next are usually performed iteratively. The goal of this phase is to perform a risk assessment of the project's estimate. Risk has been introduced throughout the process starting with the uncertainty in the design, continuing with the size and environment uncertainty, and culminating in the accuracy of reconciling the various estimating model's assumptions with the project requirements. Added to this is the basic accuracy of the model itself. Throughout the process, we have captured uncertainty information and we use that in this phase to quantify the risk and present the resulting information to management in a manner that facilitates decision making in establishing the budget or pricing for the project. Adding hardware and services to this phase adds additional sources of risk to be analyzed without changing the basic process.
Phase 7 – Budget/Pricing
The goal of this phase is to reconcile the estimate, the risk, and the business conditions prevalent in order to determine the actual budget to be allocated to the project or the proposed price to be bid in competitive situations. Management's job is to understand the estimate and associated risks sufficiently to set the final budget. Once a budget has been decided upon it will require a round of risk analysis because of the difference between the estimate and the budget is another source of risk that must be considered. The manager's job is not changed with the addition of hardware and services to the process.
Phase 8 – Dynamic Cost Estimating
The last phase in the original process was intended to provide the project manager with decision making information throughout the development project. It required tracking all the inputs (design, size, and environment changes) as the project progressed and continuously re-estimating the project over time. This activity supported earned value approaches and provided estimates-to-complete that were based on actual progress with a parametric basis. Means are available to re-calibrate parametric models on-the-fly during the course of the project and accurately predict changes to the total effort/schedule quickly compared to the sometimes laborious effort required for earned value type methods. The last goal of this phase was to capture the actual size, effort, and duration in a database as historical data to be used in future estimates. Adding hardware and services will require expanding the database schema and design to accommodate the extra types of components.
The demonstration consists of a threaded scenario illustrating estimating throughout the lifecycle of a project. The demonstration starts with a project description and asks if a given budget/schedule would be adequate. It then assumes that a project bid has been received and asks if the proposed price is reasonable. The demonstration then moves through several situations concerned with the level of staffing and the skill levels of available personnel. The demonstration uses the DOD‘s public domain Revic model to illustrate the estimating techniques throughout the project life cycle.
Kile, R.L. (1986, May). A Process View of Estimating. Revic User's Group Meeting, Albuquerque, NM.
Software Engineering Institute (2002). CMMISM for Systems Engineering/Software Engineering/Integrated Product and Process Development/Supplier Sourcing, Version 1.1 (CMMI-SE/SW/IPPD/SS, V1.1) Staged Representation; CMU/SEI-2002-TR-012 Carnegie Mellon University.
Boehm, B.W. (1981). Software Engineering Economics, New Jersey:Prentice Hall Inc.,
Proceedings of PMI® Global Congress 2003 – North America
Baltimore, Maryland, USA ● 20-23 September 2003