Earning the right to earned value
seeking EV nirvana
Even though the concept of earned value has been a mainstay of the United States Department of Defense (DoD) for many years, even though there are a plethora of opportunities to learn about earned value, even when consultants advise the “states” within the United States that earned value is an ideal way to track projects, states have not yet made a wholesale adoption of earned value in their projects. Though state governments have made great strides to better manage their projects, many states have not even so much as run a pilot project using the earned value concept.
The challenge of using earned value has been thwarted by a number of things including lack of access to needed data, the nature of funding of state projects, and just the resistance to change.
Though states within the United States may not use earned value there are many efforts to improve management of projects that deserve credit. In order to put the situation in perspective, let's review the metrics that state governments use.
Putting Things in Perspective
There seems to be a propensity to compare ROI and Deming techniques to earned value. The various techniques all have a place, but they are not comparable measures. They are not intended to look at the same issues. In order to better understand the use of various metrics, take a look at Exhibit 1.
Though the axis in the diagram do not represent true variations along the axis, the diagram does make it much easier to identify the challenges in earning the right to earned value. The vertical axis separates operations from projects. The horizontal axis separates an emphasis from effectiveness to efficiency measures.
Let's take a look at each of the areas and the metrics involved.
Ongoing Operations Efficiency
Virtually all states have earned the right to partially measure ongoing operations efficiency identified as the first quadrant (operational efficiency). In their major operational areas, states easily give some operational statistics. A Nebraska example of the monthly Medicaid payments is given in Exhibit 2.
The reason for stressing “partial” right to measure is based on two factors. First, most states measure the associated data processing costs in one area and measure other operational costs in another area. Second, the availability of federal funding for the projects associated with the operational areas cause a new definition of project verses operational costs. As professional project managers, most PMI members believe that a project is completed when the requirements for the project have been meet, the system tested and put into operations. There is a tendency for the projects to be declared complete when all features required to maintain federal funding have been met. Then subsequent features originally included in the project plan are added as part of system maintenance. This gives a false operational cost. The federal funding and possibilities of sanctions (withholding of ongoing funding) for uncompleted projects can change the definition of project completion.
The larger states in this study had progressed further in having processes to track total costs, not just IT costs. Caltrans is the California State Highway agency. According to Nigel Blampied, Caltrans keeps extensive records of operational efficiency (vehiclehours of delay, pavement condition, crash records— numbers, locations, fatalities, etc.) According to Karen Son, the state of Texas also has processes in place to insure the entire cost of a service was tracked. Nebraska is in the process of installing an enterprise management system. In the future, Nebraska may also be able to enter the ranks of complete cost tracking.
The Institute of Medicine published The Future of Public Health in 1988. In the book, the public health industry was taken to task for reporting the wrong things. The authors emphasized the need to measure the effectiveness of the public health programs, NOT the efficiency with which certain tasks were done.
A recent governor of Nebraska had a train analogy to keep spending down. He ask the various agencies to cut off the cars at the end of the train when new legislation dictated the need to add “new” cars. Great for legislators who could keep asking for new programs. Terrible for the agency heads in deciding which cars to cut at the end of the train. Yet the impact was to direct attention to the effectiveness of programs and to cut programs that were not effective.
All states have to deal with the cutting of old ‘cars’ off the end of the train. This can be a very political process in some states. Many times the work is left to the agency heads with various supporting groups making the argument to leave their “car” alone. But some states do better.
The challenge is measuring the effectiveness of the various systems. This requires an ongoing process. The easy measure is the efficiency measure. The answer that a state dispensed one million dollars for a mere cost of one hundred dollars is not a measure of effectiveness, though the argument mistakenly used when funding is challenged.
Panel of Experts Choose
A popular approach among states is to have some formalized panel of experts to guide the IT expenditures. These formalized experts generally do NOT deal with the ongoing operational systems, but rather with new systems. The state of Washington uses a portfolio basis to merge new projects into the current mix of operating systems. The California State Highway agency has multiple groups to help decide priorities. Kansas has a formal group called the Business Technical Council, which is made up of business and IT staff and helps set priorities for new and ongoing projects along with setting priorities for the use of IT resources. Nebraska has a technology council as well. The council in Nebraska experiences some difficulty dealing with the very large agency proposed projects in the same mix as the smaller agency projects.
According to Stan Dittereline, Washington state bases their Information Technology plan on a portfolio approach. This is being augmented with a Core Systems Framework. The formalized portfolio approach Washington uses is defined in www.wa.gov/dis/portfolio. The Core Systems Framework is a new collection of documents that can be found at: www.wa.gov/dis/isb/coresystem. The Framework provides additional tools for helping agencies assess the business value of their core systems as well as determining what course of action to take regarding a system (e.g., retire, do noting, modify or replace). This has become a major issue for Washington, which has some old legacy systems which are neither efficient or effective.
In the California State Highway agency, the main source of funding is the State Transportation Improvement Program (STIP). STIP funds are divided by formula between the 58 counties. The Regional Transportation Agency for each county decides how funds are allocated to projects. RTPA Boards decide which car they cannot afford. The secondary source of funding, the State Highway Operation and Protection Plan (SHOPP) is dedicated to State Highway safety, repair and rehabilitation. Funds in the SHOPP are assigned to projects through needs-based formulas.
Priority Setting Approaches
Bruce Miller, PMP, made a presentation at the PMI Sept 29-Oct1, 1997 Symposium that gave a form to establish priority setting. Bruce stressed that his illustration of the technique was just an illustration and that each organization should pick the parameters important to them. In the illustration, there were four categories for the first level breakdown of parameters. Two of the four were truly effectiveness measures. The first area included customer and management support. The second category measured the strategic fit for the organization. But the last two categories were more aimed at efficiency issues of ability to handle the project. Though the presentation stresses that the illustration was only that, it appears that the illustration parameters fit reality closely.
The challenge is to pick important projects, not projects that can be done. If 50% of the choice is based on the planning that shows that the project can be done, then a state should change the process. If a project that shows promise is not well planned, the solution is to send the project team back for better planning, not the elimination of that proposed project.
Again the larger states surveyed had more formalized approaches. There was less chance that projects were picked just because they were better planned. The larger states had a process that guided them in picking projects to plan well.
Classically, the measure for the efficiency of a project has been some combination of schedule, cost and performance. But these measures are at best cumbersome. What about a project that is under schedule, but over cost? How about under cost but over schedule? Are these projects in good shape or bad shape? Concepts such as earned value provide a much better measure of project efficiency.
Earned Value approaches have been used by the Department of Defense since the 1960s. For an outstanding history of the used of earned value, look at the paper “The Earned Value Body of Knowledge” at URL http://www.suu.edu/faculty/christensend/Pmi99qf.pdf. The concept is now becoming of age for state governments. The state of Washington substitutes the use of fixed cost contracts for the need for project efficiency measures. The California State Highway agency has a history of publishing Earned Value data on all major projects on their Intranet (internal). EV data will be published for projects exceeding $750,000 on the Internet (for the world to see) in April 2002. Michigan has progressed to the point that they have Earned Value specifically identified in their documented PM principles and has put those concepts to work in some pilot projects. Kansas and North Dakota expressed interest in the possibilities of implementing Earned Value in their states. Nebraska is including EV in one of their complex projects and attempting to use earned value in another project.
The state of Washington has eliminated some of their need to measure project efficiency. The primary reason is that they use Fixed-Price contracts with contractors who are developing new systems or integrators who are supplying major systems to the Washington state. Applying lessons learned from major IT projects that failed in the early to mid-90s, Washington State has developed fairly rigorous and sophisticated contracts that clearly define deliverables through our contracts and statements of work. For each deliverable that has a payment specified in the statement of work, there is predefined acceptance criteria that specifies the conditions that make the deliverable acceptable to the state agency. Only when the deliverable is accepted, does the vendor receive payment (minus retainage which is usually 10%).
But for most other states that retain more control (and thus risk) on major projects, project efficiency measurement is a key issue. A statement in Michigan's “Project Management Methodology” at their website www.state.mi.us/cio/opm reads:
“Performance measurement is used by agencies when they adopt techniques such as Earned Value Management (EVM) to help assess the magnitude of any variations that do occur. An important part of schedule control is to decide if the schedule variation requires corrective action. Performance measures provide some external ruler to assist management in making a decision on more than just a gut feeling about the significance. For example, a minor delay on a noncritical activity may have little effect on the overall project, while a much shorter delay on a near critical activity may require immediate action.”
Texas adheres to some strong reporting against goals (quarterly) even though earned value has not used. Nebraska as well as other states have some reporting requirements. Nebraska has a quarterly report for all major IT projects. Texas reports were used at a high level that the current Nebraska reports.
Though several states have expressed interest in the earned value approach, few have been found that actually use it. California State Highway agency uses earned value as part of contract management. The State of Michigan has used earned value on several projects, including the eMichigan Portal Implementation project. According to Dan Buonodono, Michigan is gaining knowledge in the area of earned value from a number of projects, and recognizes the need for its use. Michigan's goal over the next 6–18 months is to include at least high-level EV measures on all IT projects. Michigan is currently using earned value on some larger consulting engagements—where Michigan tracks budgeted hours verses actual hours.
Nebraska has some attempts under way to use earned value.
Earned Value Progress at the State of Nebraska
A major project for Nebraska is a replacement of the old accounting and personnel systems with an enterprise management system. Though the vendors delivering the system have a fixed cost contract, the earned value approach is being used. Federal funding issues do not exist for this effort, so Nebraska is more able to treat this project much the same way that any other enterprise would. The project had to be rescheduled recently and the project manager was ask to comment on the usefulness of the earned value approach. He stated that:
1. The Steering Committee has expressed some concern about the concepts. I think this is because they are new to the EV measures. Few will say they are comfortable as of yet.
2. We're not a “perfect” example because we're a fixed bid project. Skip (Philson, Project Office) can talk about how we've accommodated this aspect with EV.
3. The CPI tells me we've closed several tasks after expending fewer hours than had been planned. This allows me to question if we've simply misestimated or if we're closing tasks before completing their scope of work. I don't think I would have picked this subtlety up without EV metrics.
4. The EV metrics also helped me identify problems with the project plan that have led us to replan some of the phases. The replan would have occurred without EV but the metrics helped confirm what I was observing in human behavior.
5. We don't “earn” any value until the task is complete. I prefer this to the “% complete” estimate method—I think it paints a more conservative picture of our progress (and highlights the impact of late tasks).
A second major project to update an existing system is being tracked using earned value with a completely different outcome. The project is nearly half over; the critical ratio has been outside the acceptable range since the start, yet the project manager is ignoring the results based on his knowledge of the situation.
In the third case, another agency besides HHSS was ask about the desirability of using earned value. The answer was a resounding “we are not interested.” Though Nebraska would like to brag about their efforts with earned value, this approach still has limited acceptance.
It took Nebraska's Project office about six months to experiment and put in place the earned value tracking for a major project. The first decision made was to use hours, not dollars for the cost equivalent tracking. There is a disparentcy between the cost associated for contractors and the internal personnel who could alternately be working on the same task. The alternative could have been to use an average hourly cost (a practice used by some private industry companies when doing earned value). Also, Nebraska is tracking only the hours of the developers and not other costs within the earned value model. This is justified on the basis that most of the costs for this development is hourly charges.
The second decision was to drop some of the ongoing tasks that have traditionally been part of the tracking of projects in Nebraska. Such tasks as project manager time typically assigned to one long task, cultural change management, which is an effort to train the state in the upcoming changes, and any other long duration task that did not reflect the project progress toward completion.
The third decision was to export the data to a spreadsheet from the Niku project tracking system. Niku has some built in earned value analysis. This internal analysis was used as a crosscheck to the spreadsheet approach. The spreadsheets allowed more control on display of results and removed restrictions of Niku task tracking of the long tasks. Since earned value is new to Nebraska, the ability to color code numerical charts worked better that the graphic outputs usually associated with earned value. Exhibit 3 is a sample of the charts used by Nebraska.
Nebraska found that it was necessary to change some aspects of the traditional task breakdown practices of the past. The strict adherence to activity breakdown hour limited to 40–80 hour endeavors were critical to allow the earned value to be done on a weekly basis. Nebraska used the recommended approach of counting work only when the activity is complete. All workers had to clean up time reporting and input hours weekly.
With this level of detail, Nebraska found it better to group tasks to be completed in the same general time frame to better allow roll up of the earned value. At the same time, the necessity of strict dependencies with this level of planning was not as critical. Of course the project had to be baselined and estimated properly. One of the unusual result of the exercise is that the project manager was able to observe that the estimations were high.
Judgment on why the earned value approach did not give believable results in the second trial case is still out. From the successful case, Nebraska knows that the project manager was a strong supporter of the process and very involved in making decisions on the earned value approach. The effort to set up the first earned value tracking took a lot of time and thought. The same level of effort may have not been included in the second project.
Nebraska is well aware that earned value tracking is a cultural change and that not all Nebraska agencies are yet ready to get involved. There are a lot of preparations necessary to make the process work. Yet, the project office within Nebraska believes that the federal government agencies outside of DoD will make future pushes to include earned value reporting on federally funded projects. Nebraska will be ready when that happens.
Washington State Website: http://www.wa.gov/dis/portfolio/ Has the documents that explain portfolio management as practiced by the state of Washington.
The Future of Public Health, Institute of Medicine, National Academy Press, October 1988.
Texas State Website: http://www.dir.state.tx.us/eod/qa/bestprac.htm Used as a starting point for this study.
Bruce Miller, PMP,PMI Sept 29-Oct1, 1997 PMI Symposium presentation, Project Management Services, Origin Technology in Business.
Michigan's “Project Management Methodology” at website www.state.mi.us/cio/opm
The Earned Value Body of Knowledge at website: http://www.suu.edu/faculty/christensend/Pmi99qf.pdf Originally presented at the 30th Annual Project Management Institute 1999 Seminars & Symposium (October 10-16, 1999).
As true with any paper that claims to shed light on what others are doing, a number of kind people had to take time out of their busy schedule to prove information. A special thanks to:
Nigel Blampied – California
Cheryl Weber and Loren Benoit – Kansas
Dan Buonodono – Michigan
Roger Hertz – North Dakota
John Cuddy – Oregan
Karen Son – Texas
Stan Ditterline – Washington State
Skip Philson, Tom Conroy & John Ogden – Nebraska
These people all had direct input to the paper. Any shortcomings or reporting errors are my own.
Proceedings of the Project Management Institute Annual Seminars & Symposium
October 3–10, 2002 • San Antonio, Texas, USA