EVM on service projects-- an optimized paradigm
EVM has traditionally been put aside for service type contracts. Even the Defense Contract Management Agency (DCMA) will exempt very large maintenance projects from OMB Circular A11 requirements. But why throw out the benefits of EVM when you can capitalize on the valid components the project does have left and can replace or reverse engineer the components that are suspect? This paper will demonstrate very viable and beneficial methods to achieving the proven benefits of EVM for your service contracts.
Service projects have regularly set aside Earned Value Management (EVM) because of the inherent Level of Effort (LOE) scope involved. LOE effort does not appear to provide Earned Value Management benefits in a traditional left to right view. Earned Value or Budgeted Cost of Work Performed (EV or BCWP) for Level of Effort work by definition is equal to the Scheduled or Planned Value Budgeted Cost of Work Scheduled (PV or BCWS), which eliminates the Schedule Variance. In addition, Cost Variances are not truly representative of the operational effectiveness in the project. Earned Value Management just doesn’t seem to have any value for this type of project, but is this really true?
The Office of Management and Budget (OMB) requires EVM for all capital acquisitions and other major acquisition programs involving development (i.e., project) effort. This requirement applies to all contract types, without exception. OMB also recommends that EVM be used on other than development contracts whenever possible. Under the current OMB rules, programs falling short of their cost, schedule, and technical objectives may risk reductions in funding or even cancellation; EVMS is required to track program progress to these objectives. (OMB, Circular A-ll, 2007)
But the problem with ascertaining a significant performance evaluation from service contracts has resulted in almost all government agencies spelling out exceptions for LOE work.
The Federal Aviation Administration’s Earned Value Management Guide calls for a set-aside for Operations and Maintenance projects. “Operational and steady-state programs or useful segments funded by Operations and Maintenance (O&M) funds are not required to use EVM, but may do so when the nature of the work lends itself to earned value performance measurement.” (FAA, Section 2, p 8, 2010)
The Department of Defense in section 18.104.22.168.3 of its Earned Value Management Implementation Guide states “The application of EVM to contracts that may be categorized as “non-schedule-based”, i.e., those that do not ordinarily contain work efforts which are discrete in nature, should be considered on a case-by-case basis.” (DOD, Section 22.214.171.124.3, 2006 p12-13)
NASA, in its revised acquisition policy 1834.201 Policy, Clause (iv) states “EVM is not required on contracts for non-developmental engineering support services, steady state operations, basic and applied research, and routine services such as janitorial services or grounds maintenance services. In these cases, application of EVM is at the discretion of the program/project manager.” (NASA Procurement Notice 04-19, 2006)
The Department of Energy in its Applications Guide specifies that “Implementing Earned Value Management on a project, where the work is primarily categorized as Level of Effort (LOE), may not be of any significant value for managing the project because Earned Value on LOE simply measures the passage of time. There is no relationship to the actual performance on a project because value is earned as it was planned. That is, there is no schedule variance because BCWP (Earned Value) is always equal to BCWS (Planned Value) and is not tied to specific deliverables.”(DOE, Part 1, Clause (B), 2005)
The list goes on, with virtually all agencies calling for either an outright exemption or program manager discretion, which is usually determined by how much of the project work is discrete. But the catch in all of the guidelines is that the project is not off the hook and there will be periodic reviews\evaluations of service project performance. Quality performance will be reviewed first. This is usually a formality in that quality reviews are an ongoing part of the process and are formally reviewed by the customer on at least a monthly, if not weekly, basis. On the cost side, unfortunately, a majority of these reviews occur at the end of the period of performance. This can be quite late in finding out if the project is performing acceptably in the customer’s eyes.
On the contractor side of the project, the National Defense Industrial Association, Program Management Systems Committee has a standing group that is continually looking at the service project question. The NDIA PMSC EVMS Application Guide says that “Where work is discrete, EVM can be effectively employed. Where work is T&M/LOE, performance must be evaluated using other means that are not related to schedule milestones or measurement of progress.”(NDIA-1, P21, 2007)
In its publication, Services Contracts and Earned Value Management (EVM), the NDIA documents two standard approaches to service projects
“Perhaps the most common approach, when EVMS is mandated, is to treat the services component as Level of Effort (LOE). This is especially true if the resource levels are fixed. Using the LOE method to measure performance on service work leads to several problems, and is one of the primary reasons that the PMSC began exploring the topic of applying EVM to service contracts. These problems include:
- A lack of visibility into the performance of the service effort.
- A distorted view of summary-level performance due to LOE offsetting unfavorable performance variances/trends, which may be occurring on the development or other discrete component of the work
- Inaccurate performance/progress reporting resulting from work performance being defined in ways that are contrived and not well supported by existing EVM systems.” (NDIA-2, p 3, 2010)
“Another common approach is to ‘projectize’ those elements of service work where discrete output can be identified and/or scheduled. This method involves driving down to lower levels of task detail in order to find measurable completion points. The advantage of this approach is that it is often possible; nearly any recurring job is made up of many small process-oriented tasks that themselves have discrete start and finish points and identifiable completion criteria. Production runs of similar units or other repetitive service work, where similar work effort is delivered in planned quantities, can be successfully measured using EVM techniques. The disadvantage is that the planning and monitoring effort associated with managing a large number of small, non-repetitive or otherwise highly variable tasks is generally not cost effective.” (NDIA-2, p 3, 2010)
So, it appears that we are left with an ineffectual solution or a solution that in most cases is not cost effective. Well, by taking a new look at this old situation, the following sections will shed some new views on this thinking and demonstrate that EVM can still be an effective project management tool, even on service projects.
This paper will show two approaches to deriving value from EVM for service-type projects. The first will be a new look at the LOE approach above. In a right to left interpolation of the traditional EVM equations, the re-emphasis of the utilization of the EAC will be presented. The second will focus on some variations of the “projectized” work order\job order breakdown approach. Tools and techniques will be presented to demonstrate beneficial, cost- effective results.
The goals of any project are to finish on time, within budget, and with all deliverables meeting customer satisfaction. Service projects are no different. Off the top, however, they will always finish on schedule, because the project starts on the first day of the Period of Performance and ends on its last day. If this work is categorized as Level of Effort (LOE), then the Schedule Variance (SV) is always 0 and the Schedule Performance Index (SPI) is always 1. With Standard EVM, the cost performance side of the equations are thrown back into a cost versus budget relationship. This, though not a totally useless calculation, has a tendency to hide both poor and outstanding performance areas. Even with schedule out of the picture, the relevant cost questions that EVM answers are still valid. How much will the project cost, how well is the project performing and, if applicable, how well do I have to perform in the future to bring costs back in line?
Right to Left View of EVM Calculations
The first approach takes a non-standard right to left view of the standard EVM calculations. In this, we will start at the end and work backward.
The first step is to determine a detailed and accurate Estimate at Completion for the project. The detail level should reflect the WBS setup for the project and at a minimum be able to answer any questions on funding status. It should reflect all costs that are being charged and will be charged to the customer in the funding categories required. This will be a ground up effort based on the EVM equation: EAC = Actuals + Estimates to Complete (ETCs). If required, the EAC model should be able to take into account the accurate allocation of non-productive support costs, such as project management or site management. Actuals should be updated weekly and ETCs should reflect anticipated weekly expenditures for both labor and non-labor costs.
All costs should be included. Prime labor as well as subcontract labor, direct costs as well as indirect costs and fees. In laying out this model, actual names of persons and their costs in each shop should be avoided for flexibility and personal security. Position titles using average or composite rates should be used. This model lends itself to changes in shop assignments and changes in vacation schedules. The resulting model will also lend itself to what if analysis and prototyping.
A side benefit, at least for government projects, is an accurate, defendable model for funds status. This gives the contracting officer optimum time to request and or adjust funding and results in the reduction if not elimination of working at-tisk.
In the second step, we reverse calculate the cumulative EV based on our constructed EAC, using the formula
EV(cum) = (Budget At Completion X Actuals To Date)/EAC
This value can be used in EVM to determine the cumulative cost variance (CV) and the cumulative cost performance index (CPI).
The third step would be to calculate a period EV by subtracting the last period EV(cum) from this period’s EV(cum) The result would be an interpolated Earned Value or performance for the period. This value can be used in EVM to determine the current cost variance (CV) and the current cost performance index (CPI).
In the beginning, use your percentage thresholds only to identify variances that need to be analyzed and possibly corrected. As you gain confidence in the accuracy of your EAC, you may bring the dollar threshold into your analysis. An indication of the accuracy of your EAC is that your ETC for the period is consistently within plus or minus 5% of our actual for the period.
Finally, conduct a standard EVM cost analysis. Determine the variances to act upon and develop appropriate corrective action plans. Trending the current CPI and the cumulative CPI will give you a very good picture of performance and even in an excellent shop will spot complacency and or scope creep.
Three notes must be made at this point. The first is DO NOT use the calculated EV numbers in any EVM schedule calculations. The results will not make sense and have no notable management value. In addition, you don’t have to worry about the schedule anyway. The second is a recommendation. Set your initial percentage thresholds low (e.g., 5%), determine the number of relevant variances to be address, and adjust upward in agreement with project management. The last is a double leap of faith. Because your LOE calculations are skewed already, use the calculated EV from this approach in your EVMS. You can then capitalize on the variance reporting functionality of your system. And, last, since you are putting the effort into building and maintaining an accurate EAC model, also use the EACs and ETCs in your EVMS.
This approach will let the project realize the relational cost control benefits from EVM while addressing the following problems stated in the introduction:
- “A lack of visibility into the performance of the service effort.” The above approach, if the EAC is modeled accurately, will give visibility into service performance and variances.
- “A distorted view of summary-level performance due to LOE offsetting unfavorable performance variances/trends” Using a more accurate EV than LOE presents we have a clearer view of both detail level and summary-level performance
- “Inaccurate performance/progress reporting resulting from work performance being defined in ways that are contrived and not well supported by existing EVM systems”; well, we might still be contrived but we are a lot more accurate in our contrivance.
“Projectizing” Service Work
“Production runs of similar units or other repetitive service work, where similar work effort is delivered in planned quantities, can be successfully measured using EVM techniques.” (NDIA-2, p 3, 2010) If the work does not fit these bounded criteria then the principle “disadvantage is that the planning and monitoring effort associated with managing a large number of small, non-repetitive or otherwise highly variable tasks is generally not cost effective.” (NDIA-2, p 3, 2010)
The second approach keys on eliminating this huge amount of work, while maintaining most of the benefits. Instead of the projectized work order level, we can look at the next level up, (e.g., the shop level), and then consolidate work order performance at that level. If we use readily available performance measures usually provided by quality and or production control we can eliminate a great amount of work.
The second approach keys on service performance. The focus here is to determine “quantifiable” metrics of service performance, establish a baseline, and then use EVM to show the variances. The key is quantifiable performance. The NDIA PMSC Service Project Committee puts it very succinctly.
“Budgeted Cost of Work Performed (BCWP), or Earned Value (EV), is the key EVM performance measure, and can be applied any time you want to compare an accomplished quantity of work to a planned quantity of work. This method of tracking performance works equally well for measuring a series of individual deliverables, as in a project, or when measuring the production of units within lots, as in a manufacturing environment. Some service efforts resemble manufacturing lots in that they provide a recurring or predictable quantity of work; these services may also be measured using BCWP.
BCWP does not apply to service work where the performance is being measured in terms of quality rather than quantity. Common examples of qualitative service performance metrics include availability and reliability. A service metric that measures the percentage of time that email is available to users during standard working hours, for instance, does not consider the number of emails sent. Similarly, a metric that tracks mean time between failures does not reflect the absolute number of failures during the reporting period.
One way to help decide if BCWP should be used to track performance is to determine whether or not the performance can be made up in a later period if it falters in the current period. When performance on a project or in production of a manufacturing lot falls behind, the BCWP will be less than the budgeted cost of work scheduled (BCWS). Assuming the project or product run is not cancelled, the work will eventually be made up and BCWP will equal BCWS when it is all completed. Much of EVM analysis relies on this truism: that BCWP equals BCWS when the work is completed. In the case of a qualitative metric, however, there is no making up for missed targets in later periods. If the target for email was that it be available 100% of the time during normal business hours during the month, and it was only available 90% of the time, there is no way to make it available 110% of the time next month and bring the variance back down to zero. Over time, if BCWP is used as a proxy for qualitative service metrics, variances will inevitably accumulate, and the program will show constantly increasing cumulative variances that (1) must be explained in a formal report if you are using an ANSI-748 EVMS, and (2) provide no meaningful management information.” (NDIA-2, pp 6–7, 2010) (ANSI, 2009)
So, we have our criteria: quantifiable, recoverable, cost effective, and meaningful. For service projects these metrics should reflect service performance. Some prime candidates would be man-hour utilization and percent productive man-hours; taken together, they give some indication of active service performance and the project impact of support or non-productive hours. They both reflect cost performance and are readily available and accurate because they are usually reported to the customer on a regular basis, and the steps to using these are as follows.
First, establish a WBS that reflects work broken down to the type of service and the shop or site. This breakdown should coincide with the level at which quality metrics are being analyzed and reported. This level is usually dictated by the customer, but may take a meeting of the minds between quality\production control, project management, and the EVM analysis group to finalize it.
Second, establish a meaningful baseline metric, based on the acceptable standards set at the beginning of the project. The project was given goals and acceptable quality and service metrics up front. Although quality is an encompassing metric, it is also an assumed entity in EVM calculations. You can attempt to bring in pure quality performance metrics but they will only cloud the issues as far as cost control is concerned. For example, if the acceptable level of man-hour utilization is 90% and the goal is 95%, pick either metric as your baseline compare. It does not really matter in that the baseline metric you choose is for comparative purposes. It will be the denominator in your calculation.
Next, calculate a service performance EV for the reporting period, based on your chosen metric(s). The calculation will be as follows:
Period EV service = (Period Metric X Period Budget)/Baseline Metric
Add this value to your cumulative EV from the last period to get a new cumulative EV.
Now, do period and cumulative EVM cost calculations and your variance analysis; agree upon percentage thresholds for variance reporting and hence project management response. Dollar thresholds may be meaningless or misrepresentative for these calculations.
As in the last approach, there are a couple of things to note. Again as in the first approach, DO NOT use the calculated EV numbers in any EVM schedule calculations. Next, multiple criteria may be combined by multiplying them together. Multiplying the man-hour utilization by the percentage of productive man-hours yields a more stringent project performance calculation. Finally, and very importantly, the EV calculated here is for COMPARATIVE purposes only. Virtually all of these metrics are sensitive to service disasters or outliers that can move the metrics to a point of not representing the basic performance of the service. On a positive note, the cumulative calculations will tend to normalize any outliers to some degree. On a final note, do your root-cause analysis on the period calculations but base the manpower change considerations on the cumulative variance analysis.
Both approaches have been proved to be of indispensible value to a large U.S. Army vehicle service project in Iraq. This project covers 12 missions at 8 sites, with 7 of these missions involving wrench-turners. The entire project is LOE, except for a few isolated repair or installation campaigns, representing less than 2% of the project value. The project uses the first approach on a weekly basis and integrates the month end calculations into the EVMS reporting. The project is very incrementally funded. It has had 30 plus contract modifications, representing 100 plus funding changes, yet has never worked at risk.
The second approach is also calculated on a weekly basis and is used as both a reality check of the first process and additional information for root cause determination of service variances. Both combine to assist program management in making optimum manpower and facility changes.
The customer is very satisfied and the project has become a standard that the customer compares others with. Both Contract Performance Appraisal Reports to date have given the service effort very good to outstanding ratings for each of our missions. The result has been the awarding of two additional large task orders to the original scope.
No methodology or process will make projects profitable overnight. Combining EVMS principles and performance indicators provides you with early warning signals, which allow you to apply effective corrective action. EVMS helps you reduce risk in a project’s outcome. This is true for all projects, including service-type projects.
Although government agencies recommend the exemption of service type projects from EVMS requirements, there is no need to set aside this valuable tool. By taking a new look at this old problem we can again use the value in Earned Value Management.
I have presented two of these reconsiderations. The first, although the most robust, can actually be of indispensible value to any type of project, but was developed to extract the most value for service types of work. The determination of an accurate Estimate at Completion (EAC) is a foundation feature of EVM. When a service type contract hinders that determination we must resort to other means. Without a doubt, the most accurate EAC is built from the ground up. Estimates to Complete (ETC) are added to the Actuals to Date to give us our best representation of what the project will cost. To construct this EAC we must specify a building of ETCs that represent our best plans known at the time. All expected costs should be determined and planned in weekly buckets to the end of the period of performance. With an accurate EAC established we can do a right to left calculation to determine a valid Earned Value for each control account. The cost control analysis can then proceed as normal.
The second approach actually preceded the first in its conception. In looking for alternatives to disregarding EVM for a large service project, we looked at available and accurate service performance metrics that would fit the criteria needed to be performance indicators. We needed metrics that were quantifiable, recoverable, cost effective, and meaningful to not only to service performance but to the project as a whole. Our initial choices were Man-hour utilization and percentage productive man-hours. Multiplied together and compared with the original metrics specified by the customer, we were able to calculate a more representative earned value than the default offered by standard LOE work. This worked very well for months, but we needed more, so we invested in the second more accurate approach. We still use the performance metric calculations as a reality check and an additional tool for service variance analysis.
The EAC model approach has already been “productized” and is presently being beta tested on two other projects. Interfaces are being developed for the three principal EVMS systems being used in the company, and a CFSR draft generator is being tested for projects that require it as a deliverable.
By not abandoning EVM on our service project and by giving it some new looks we were able to get all of the cost control benefits offered by Earned Value Management.
American National Standards Institute/Electronic Industries Alliance (ANSI/EIA)-748b (2007), Earned Value Management System, Retrieved from http://webstore.ansi.org/RecordDetail.aspx?sku=ANSI%2FEIA-748-B
National Aeronautics and Space Administration, (2005, November 13) Procurement Notice 04-19, 11/13/2006. Revised Acquisition Policy 1834.201 Policy, .Clause (iv),
NDIA (2007, March) ( National Defense Industrial Association (NDIA) Program Management Systems Committee (PMSC) Earned Value Management Systems Application Guide, March 2007. Retrieved from http://www.ndia.org/Divisions/Divisions/Procurement/Pages/Program_Management_Systems_Committee.aspx#documents
NDIA (2010, February 17) National Defense Industrial Association (NDIA) Program Management Systems Committee (PMSC) Services Contracts and Earned Value Management (EVM), February 17, 2010 Retrieved from http://www.ndia.org/Divisions/Divisions/Procurement/Pages/Program_Management_Systems_Committee.aspx#documents
Office of Management and Budget (2010, July 21) Circular A-11 (Part 7, Planning, Budgeting, Acquisition & Management of Capital Asset), Retrieved from http://www.whitehouse.gov/sites/default/files/omb/assets/a11_current_year/a_11_2010.pdf
U.S. Department of Defense (2006, October) Earned Value Management Implementation Guide, Retrieved from http://www.srs.gov/general/EFCOG/02GovtReferences/01DOE/DOEApplicationGuide.pdf
U.S. Department of Energy (2005, January 1) Earned value management application guide version 1.6 Part 1, Section Risk Based Applications Clause (B). Retrieved from https://acc.dau.mil/CommunityBrowser.aspx?id=19557
U.S. Federal Aviation Administration (2010, January) Earned value management guide, Section 2. Retrieved from fast.faa.gov/docs/EVMGuide.doc
© 2011, Allen H. Smith
Originally published as a part of 2011 PMI Global Congress Proceedings – Dallas, TX