The EVM maturity model--EVM3™
Rapid and widespread acceptance of Earned Value as a “best practice” has encouraged an increasing number of institutions, both in private and public sectors, to include Earned Value in their project management process. Yet, often the first attempted use of Earned Value results in frustration and abandonment of Earned Value methods. How sad. Any further attempted use of Earned Value may be prevented with “we tried that once, already” corporate mentality. The author believes, without a statistical sampling, that these first attempts to use Earned Value attempt to do too much too soon. Presently, there is no guidance on how to grow an Earned Value capability from nothing to a fully ANSI/EIA 748-98 compliant system. Lacking this EVMS guidance, organizations look at the ANSI Standard as not only the goal, but also the first step.
The author proposes a five-step model for organizations to use in implementing Earned Value. The process, if followed to its end-point, ultimately establishes a method to continually refine the Earned Value process to improve its benefit in project management. Firms with an ANSI compliant Earned Value Management System can use this model to establish Earned Value process metrics and create Earned Value improvement plans.
How do you measure an organization's capability in applying Earned Value Management (EVM)? What is the subset of ANSI/EIA 748 requirements that are sufficient to get started in EVM? How do you know that your potential teammate or contractor has a competent EVM system in place? These questions were once easily answered when the U.S. Department of Defense (DoD) formally validated an EVM implementation using the now cancelled Government standard. General industrywide adoption of EVM using ANSI/EIA 748 and the cancellation of the DoD standard have left a void where a formal validation process once existed. In its place is “self validation,” which lacks the clarity of a formal validation review and allows for less than full implementation. Thus, customers, teammates, and senior management are left unsure about the real quality and usage of EVM within an organization. For some companies, projects, or industries it may be most cost effective to have less than full ANSI 748 compliance. This may still provide data, which is an improvement over past project management data and decision-making.
A similar situation existed in the world of software development in the ‘70s and early ‘80s. There was no way of knowing if one company or organization was better than another at software development. Yet some firms seemed, on average, to be more successful. With the rising cost and risk of botched DoD software projects the Software Engineering Institute (SEI) was created. The SEI undertook a survey to determine the processes used in firms with a better than average track record. From these efforts a series of “best practices” was documented as a goal for software organizations to attain. Although not an ANSI standard, attaining these best practices is now sought after by both developers and customers as attributes of sound software development. Known as the SEI Software Capability Maturity Model (SW-CMM), this set of practices is known to take years for organizations to meet all requirements. Yet some have, and those that haven't know where they are on the road to meeting this highest capability. The SW-CMM is built around five levels of maturity, with the fifth being the highest capability. This industry standard has resulted in customers and industry knowing the skills and practices of software developers through both internal and external assessments of their software capability against the SW-CMM.
Both software development and application of Earned Value Management are process intensive efforts. Therefore, it seems that the basic five-level CMM model may be applicable to the development of a maturity model for Earned Value, an Earned Value Management Maturity Model, or EVM3.
The Five Levels of EVM3
Level I is the level of organizations with no or limited EVM implementation in place. This level provides a defined starting point for an initial implementation of EVM and does not require a large commitment of resources. Level II defines an EVM implementation, which is less than fully compliant with ANSI 748, but may be sufficient for smaller projects or for a simplified EVM implementation. The resulting initial benefits of a Level II EVM implementation are consistent with the modest investment, thus creating a positive “first use” of EVM. This encourages further use of EVM, further investment in EVM, and a goal of attaining higher levels of EVM3. Further, Level II allows for a defined, but less than full, implementation, which should provide valuable project insight at a lower cost to the (government) customer. Level III is an ANSI/EIA 738 compliant implementation. Level IV adds guidelines for measuring the quality of EVM data and introduces metrics for measuring the health of an Earned Value Management system. The highest level of the EVM3, Level V, requires monitoring the efforts to improve an Earned Value Management system. Organizations with an ANSI or DoD compliant EVM implementation can use Levels IV and V as goals for improving the utility, timeliness, accuracy, and cost effectiveness of their EVM systems.
EVM3 Level I
Level I represents organizations with no or very little Earned Value management in use. Any use of Earned Value is limited to individuals or small projects where an Earned Value advocate uses its principles in attempting to calculate SPI and CPI even though precise planning and cost data is missing. The work packages may be only at high levels, or possibly only one lengthy work package exists. BCWS is the project's original spend plan, without regard to use of short duration work packages. The BCWP is subjectively estimated. Our user may be forced to simply estimate the hours used per week for ACWP. A WBS may not exist and data is not easily rolled up, nor is there a capability to drill down to problem areas. Our user appreciates the importance of Earned Value but finds themselves in a project or organization that does not appreciate nor encourage Earned Value Management.
EVM3 Level II
EVM3 Level II represents organizations with an interest in the use of Earned Value Management and a reasonable attempt to meet some of the good practices found in the ANSI standard. These organization meet five simplified criteria (Fleming & Koppelman) of (a) defining and decomposing the scope with a WBS-like structure, (b) planning and schedule the scope, (c) budgeting cost accounts to functions, (d) maintaining a performance baseline, and (e) monitoring project performance and forecasting outcome.
These organizations realize the value of assigning work to teams via a WBS. They use a WBS to break the project work into meaningful, manageable portions, and formally assigned it to a person or department at each lowest WBS element. At each WBS level the elements are considered to have a “contract” with defined values for the triple constraint. The budget for a function may be stated in dollars or hours for labor-intensive efforts. The means of determining work completed may be subjective and rely on percent complete estimates. There may not be an easy ability to roll-up data through the WBS. No projectwide rolled up baseline may exist. Earned Value Management may occur only at the lowest level of the WBS. A performance baseline is established at least for each control cell, and only altered through a formal change process. The baseline is kept up-to-date with formal changes to the project scope and funding.
Spreadsheets capture the plan and performance data, perform Earned Value calculations, and provide graphs.
The accounting system may not have rigorous detail at the lowest level, but an effort is made to estimate the ACWP. Lacking formal timekeeping, ACWP estimates are made based on recollections of team size, competing tasks, available time, travel, and vacation time away. Likewise BCWP is based more on subjective measures than objective measures. Thus variance thresholds may not exist, but a periodic review of variances, SPI, and CPI are formally done.
EVM3 Level II attempts to meet, in spirit, the following simplified ANSI criteria (in parenthesis):
• Define the work, using a WBS (2.1a)
• Define who in the organization does the work (2.1 b)
• Establish and maintain a time-phased budget (2.2 c)
• Establish budgets for control accounts and work packages (2.2 d and 2.2 e)
• Sum of work package budgets equal control account budget (2.2 f)
• Record costs in a manner consistent with budgeting (2.3 a)
• Compute Cost and Schedule Variances (2.4 a)
• Analyze reasons for CV and SV for project management (2.4 b)
• Make project management decisions and action plans based on Earned Value data (2.4 e)
• Revise and maintain a performance baseline consistent with authorized changes to the project scope, budget or schedule (2.5 a)
• Provide a traceable history of revisions to the baseline (2.5 b)
• Preclude unauthorized changes to the baseline (2.5 d)
• Document changes to the baseline (2.5 e)
Application of Earned Value may not be widespread or uniform throughout the enterprise. There may be projects or PMOs who use Earned Value, and others who do not. Where Earned Value is used, there will likely be some inconsistency in application since no formal Earned Value directive system exists.
For smaller projects and small organizations Level II may be sufficient. It is certainly an improvement over non-EV project management. Level II provides some of the basic functionality of an EVMS, at little cost. It can be viewed as entry level Earned Value for firms with a goal of full ANSI compliant systems. This modest list of project planning and control accomplishments can illustrate the value of EVMS to a reluctant workforce and skeptical management. With its the low cost and relaxed requirements Level II can help overcome cultural resistance to EVMS. The benefits of Earned Value will outweigh the modest cost, leading to a positive first use of Earned Value. Level II can educate the organization on the principles of Earned Value and help define the policies for a future, enterprisewide, ANSI compliant “system description.” Finally, EVM3 Level II will help define the requirements for the selection of formal EVMS tools (software) for deploying Earned Value throughout the organization.
EVM3 Level III
EVM3 Level III is simply an EVMS that meets the requirement of the ANSI standard. A EVMS System Description is published and a uniform application of Earned Value is found throughout the organization. By definition, those organizations previously meeting the DoD C/SCSC are at least at Level III.
EVM3 Level IV
At EVM3 Level IV the organization begins to collect metrics on the EVMS itself. This is a separate activity from reporting project performance using Earned Value. This effort is used to determine how the Earned Value Management System is functioning as a process. The author believes the first efforts in measuring the Earned Value process was initiated by the Naval Air Systems Command (NAVAIRSYSCOM) in 1996 for their in-house application of Earned Value. The author, while assigned to NAVAIRSYSCOM as a reserve officer, developed a series of candidate metrics (Stratton). These metrics attempt to measure the quality and timeliness of the Earned Value data gathered via the Earned Value process. The metrics described below are intended to indicate the concept of measuring the Earned Value process. EVM3 Level IV requires that metrics be collected on the process, but not mandate that a certain set be used.
Value versus work package (scatter plot). This metric measures how consistent or inconsistent the project or organization sizes work packages. Variation is expected between work packages. However, very large valued work packages may indicate an unwillingness to detail plan work while very small valued work packages may indicate planning beyond a reasonable or knowledgeable level.
WBS fan-out. This metric measures the complexity of the WBS and the effort needed to roll-up the data. This is a ratio of WBS level “n+1” to WBS level “n.” Too high a ratio may indication the WBS is too shallow and too many tasks are combined in the upper levels.
Number of work packages per CAM “Can I get some help?” This simple metric looks at the CAM‘s workload in managing his or her tasks and how uniform the work load is between projects and between CAMs. Too high a workload does not provide time for proper analysis or planning, and may lead to Earned Value data inaccuracies.
Data latency “Where my Earned Value report?” This metric collects data with a goal of reducing the latency from the end of the reporting period to its Earned Value analysis. This latency must be minimized for effective proactive management.
Number of replans and budget changes. How stable is the baseline? This metric looks at the stability. Too many replans cost project resources and indicate lack of adequate planning or managing the customer.
Number of retroactive adjustments Oops!, data error. This metric measures the quality of the Earned Value data collection (accounting) process. If retroactive ACWP or BCWP adjustments are frequently needed there is a data collection problem.
Planning package lead time “Just in Time” may be good for manufacturing but planning packages should be detail planned some time before the day they are opened for work. This metric senses if the organizational expectations on planning package conversion to work packages is being met.
Percent non-discrete work packages. How accurate is your SPI? The less LOE and subjective estimates the better the data. This metric measures the percent of non-discrete work package with a goal of reducing this percentage.
Value of replans per replan Small change? Big change? Multiple small changes indicate an ongoing effort to keep the baseline up to date with authorized changes in scope, schedule, or funding. One or two large changes may indicate a backlog of small changes that should have been addressed to keep the baseline current with formal change changes.
The important feature of EVM3 Level IV is that the organization has moved from simply collecting Earned Value data for project performance, to collecting data on the Earned Value process! The author is not aware of any organization at Level IV, although some internal unpublished efforts are probably under way.
Level IV also addresses advanced applications of Earned Value. These organizations may modify BCWP with Technical Performance Metrics (TPM) (Kulick) factors to provide an alternate view of the project health. Thus, Earned Value at level IV also addresses the Scope (Performance) side of the Triple Constraint.
Level IV may also include efforts to make Earned Value data more valuable in project-to-project comparisons, as in a PMO. Comparison of projects’ SPI and CPI consider the percent of LOE on each project, and the percent complete, to make the comparison more valid. Compensated CPI (CCPI) and Compensated SPI (CSPI) (Stratton) may used to reduce the unfair advantage large LOE give to a project's SPI and CPI.
EVM3 Level V
Level V represents the highest level of achievement in implementing Earned Value. At this level plans are put in place, with budgets and champions, to improve the quality and usefulness of the Earned Value data. Goals are set for improving the EVMS, for example: reducing re-plans, data latency, and retroactive changes. Facilitators may be made available to help project planning using less LOE. Planning packages are converted to formal work packages well in advance of commencing the work. Metrics are use to track Earned Value Management Systems improvements.
The history and evolution of the SEI SW-CMM provides the Earned Value community with a road map for implementation and refinement. Initially the SW-CMM was established to record the “best practices” and establish a graduated scale for organizations to use, voluntarily, in improving their software engineering capability. Over time the customer community began to look at the CMM as a means of selecting their contractors. The higher the CMM rating, the more likely the defined process would produce good software on cost and time. From this increased customer interest, corporate interest increased further. The positive feedback of this evolution has resulted in even wider acceptance outside the DoD. Today there is a greater percentage of interest in the CMM from commercial non-DoD industries than within the DoD community. Independent firms, upon request, will provide an organization with an assessment of their CMM level. Those with higher ratings proudly state so in proposals. The result of these independent reviews becomes part of the corporate values and performance as they compete with peers for business.
The EVM3 can follow the same path. Imagine selecting a contractor or subcontractor based up their proposal and past performance. What if you knew, from an independent assessment that they requested, that their Earned Value system is Level II, or III, or even V? How would that factor into your selection? For those who long for the now discontinued formal DoD validation process, the EVM3 provides a basis for assessing a firm's EVM system.
Once the EVM3 is in place as an industry accepted Earned Value Management Maturity Model there should be an independent assessment agency. An independent firm may use the EVM3 in reviewing and awarding an organization an industry recognized “rating.” This, like ISO 9000 and SEI assessments, could provide for an independent review of a firm's EVM system. An EVM3 rating could become a criterion for selecting a teammate or contractor.
The author has proposed a five-step maturity model as a tool for organizations to refine their implementation of Earned Value. This model, EVM3, describes the Earned Value processes expected at each level, and takes Earned Value implementation beyond meeting just the ANSI standard. The highest levels produce organizations that both measure the effectiveness of their Earned Value implementation, and establish plans and budgets for continual improvement of their Earned Value Management System.
Industry would be well served by establishing the EVM3 as a model for evaluating, through independent assessment, the EVMS of organizations and companies. Only then can customers evaluate potential suppliers on their skills and demonstrated use of Earned Value versus simply their claims.
EVM3™ is a trademark of Management Technologies.
Fleming & Koppleman. (1996). Earned value project management. Upper Darby, PA: Project Management Institute.
Kulick, Kathryn A. Technical Performance Measurement Presentation, various sites, Technical Performance Measurement Associates, Inc.
Stratton. (1997, April). Metrics for metrics or measuring your earned value system. In Control Journal, 13th Annual Conf. Proceedings, 10 (1), p. 423.
Stratton. (1999). Improving SPI and CPI calculations on LOE heavy programs. Proceedings of the 30th Annual Project Management Institute 1999 Seminars & Symposium. Newtown Square, PA: Project Management Institute.
Proceedings of the Project Management Institute Annual Seminars & Symposium
September 7–16, 2000 • Houston, Texas, USA