two important EVM indirect benefits in large IT Spanish public sector projects
We present here how EVM offered two indirect benefits that were not planned for initially. The experience was carried out in two large IT projects. The contractor had a requisite the need of using EVM for monitoring and controlling the project progress. This paper describes these two real cases with a presentation of the real scenarios (including Public sector constraints); the framework in which these EVM experiences are embedded; the methodology used for all EVM activities carried out (PMO, data collection method, analysis, tools used); the pros and cons encountered; lessons learned and the two indirect benefits encountered (that on their own justified the use of EVM): A) EVM role in friendly, objective and independent contract resolution, B) EVM role in project extra cost calculation.
The experience is part of a practical experimentation on the advantages and inconveniencies of EVM utilisation in an Organization belonging to the Spanish Defence Ministry. In this Organization EVM has been used since 2001 for monitoring and controlling performance of large IT projects. The objectives of the initiative were to demonstrate internally the benefits and/or inconveniencies of using EVM, helping achieving a higher level of CMMI (Capability Maturity Model Integration) (Solomon, 2002) in the IT department and gaining practical know-how before deciding further EVM extension to other projects in the Organization (aeronautical, spatial, defence, etc.).
The particular “culture” of the Organization was an important factor to be taken into consideration. In order to facilitate the use of EVM some requirements were set in the “Technical Prescription Document”. Also the management of the projects counted with the help of a PMO (Project Management Office).
The results obtained until today are highly positive in every sense (excepting the slightly higher managing cost and the specific training need for both the Organization and the External Consultancy Companies, that to our surprise lacked the necessary theoretical and practical knowledge on this project management technique. The two indirect benefits that were not thought or planned at the beginning of the experience, justify, on their own, the use of EVM. These indirect benefits have been clearly recognised by the Management of the Organization, giving economic advantage and confidence in the system used, as it introduce objectivity in the control of projects what is translated in stronger, more reliable and long lasting relationships with the contractors.
Material and Methodology
Information on EVM can be found in the relevant literature (Gansler, 1999; PMI, 2004; PMI, 2005) In order to arrange for an historical EVM Project data base to be used in different studies and so be able to use the experience acquired we have followed recommendation in the industry (PMI, 2005) and have taken the following actions:
- Produce a Log of activities related to EVM tracking
- Archives of all the baselines of the projects, the files containing the weekly progress and weekly version of the EVM control panel (particularized with yymmddd)
- Implementing a filter mechanism to convert continuous progress reporting by the team leaders into discrete progress (with the intention of analyzing the differences of both types of reporting – curves and indices)
- Utilization of a Project Management Office so all the EVM data and results, and all the project documentation were conveniently handled and registered in the Project Dossier (meeting minutes, tracking reporting, issues, change management, etc.)
- Formal procedure for baseline review contractually stipulated, trying to maintaining the initial baseline, whenever possible, along the whole project.
We had stated the “normal” benefits we expected to obtain from using EVM (Christensen, 1998) in our projects:
- Early and precise warning of tendencies and problems.
- Precise view of the project status (cost, schedule, progress)
- Direction change base.
- Support of mutual objectives of all intervening parts in the Project (contractor, customer)
- Help for trying conduct the Project in cost and Schedule. (Estimated At Completion, trend analysis, etc.).
Nevertheless, the “indirect”, “collateral” or “non standard” advantages were not clearly defined, and presented to us a semantic problem, as the separation line between “normal” and “collateral” advantages and disadvantages is not very well defined. It was decided to catalogue as “collateral” those “benefits or advantages not foreseen at the beginning of the project tracking using EVM. This means that the normal benefits regularly cited about the use of EVM where not considered as “collateral”:
The method chosen for analysing and rating the different benefits has been made with the use of mind maps, that helped in the selection of criteria used to point out the higher rated “collateral” benefits.
The units (or scale) of the different selection criteria were:
- Concerning the savings (or economic impact) = one unit every 100.000 €.
- Objectivity = is a qualitative measure, rated as 0, 1 or 2 in function of been considered objective by nobody, the customer or contractor, or both.
- Independency = is a qualitative measure, rated as 0, 1 or 2, accordingly being considered independent by nobody, the customer or contractor, or both.
Exhibit 1. Selection Criteria
The “collateral benefits” or “indirect advantages” reported have overcome fully the economic impact limit established and both have been rated with two points in relation Objectivity and Independency. They have contributed to improve the customer/contractor relationship.
The main constraint of both cases is that the project conforms to the Spanish Administration Acquisition Law, and the Consultancy Company and Vendor Company final budget were allocated initially during the adjudication phase. These budgets cannot be changed easily and have to follow the formal administrative procedure of Modification of the Particular Administrative Clauses Document. Therefore these are fix price projects.
Clauses applied in the “Technique Clause Document” favouring the use of EVM:
- The contractor will supply, at the beginning of the Project, an histogram of the work load of each human resource, in a weekly scale, for all the Project. Such histogram should agree with the Baseline of the Project, and will serve for the drawing of the curve BCWS – Budgeted Cost of the Work Scheduled. This information can be supplied within a MS Project file
- For monitoring the Project progress, the contractor agrees to facilitate, in a weekly basis, every Friday or the last working day of the week, the actual number of man hours invested in the Project by every person participating in the project. This information, supplied in excel, will be the basis for drawing the curve ACWS – Actual Cost of Work Scheduled.
- The contractor will supply the weekly progress of all the elementary activities (the last decomposition level of the WBS - Work Breakdown Structure). This information will serve for calculating the weekly value of the BCWP – Budgeted Cost of Work Performed. The progress of each elementary task will be given in percentage form, according the following criteria:
- Tasks not started: They will be reported as 0% progress.
- Tasks started and with less than half of the task done: they will be reported as 10% progress.
- Tasks started and with more than half of task done: they will be reported as 50% progress.
- Tasks finished completely by the contractor and pending key user acceptance: reported as 75% progress.
- Tasks finished completely (and accepted by the customer): reported as 100%
- No other progress values will be permitted.
- The contractor will supply weekly values and curves for BCWS, ACWS and BSWP, for a whole project view and milestone view. This information will be supplied in electronic format.
- The contractor will supply the weekly Schedule and Cost Indices.
- The contractor will point out those tasks with an important progression deviation, when the Schedule variance Index falls below 0,90.
It is interesting to note that we have experimented the following problems:
- Though from a theory point of view everybody agreed in applying EVM in the chosen projects, it was observed some reporting problems to our weekly data collection requirement
- It was observed a kind of inertia to measure progress against plan, as if people lived happier without knowing the truth
- Lack of practical knowledge in the involved partners
- Need to train the Project Management Office in EVM data collection, data manipulation and reporting
- Need to form management on the exact meaning of the EVM curves and indexes, how interpret them. The interpretation of the “Earned Curve” was the more difficult one as there was a tendency to interpret it as something different.
- It was thought, in order to be able to extend their use throughout the Organization to train the different project managers (some of them highly experienced) in this technique.
- It was felt that a written method, that would take into account local features, would help.
- The use of EVM in fixed price type of contracts, that is one the major contracting form used at present by the Spanish administration.
- Other special conditions derived of the Pubic Sector characteristics.
- Multidiscipline mixture (external consultants, IT personnel and key users of the Organization).
Case 1 – Objective and Independent Contract Resolution
The Defence Organization. A leading well known International Consultancy Company, also based in Spain. A two year ERP implementation project with a considerable number of gaps (related to the specific needs of the Public Administration environment). More than 50 participants (between key users, IT Organization personnel, and the International Leading Consultancy Company). The 2002 Euro Milestone (with two clear project phases – before and after the Euro Milestone). A high ratio of development against standard features. The costs of the whole project for the Organization was closed during the public bidding phase.
Due to major problems arising in the implementation (as for instance inadequate software release for supporting the Spanish Public Sector market) and with more than one year delay it was necessary to cancel the contract. Nevertheless the Consultancy Company had comply the main tasks concerning the Euro Deadline (first Phase of the Project) with a certain number of gaps partial or totally uncompleted.
Resolve friendly a contract not finished due to causes of “force majeur·. If EVM had not been used, there had not been a real vision of the progress state of the project at the cancellation contract time, it had not been possible evaluate the remaining part of the project in an objectively way and with the mutual acceptance by all participating parts without any problem. The final contractual modification agreement was reached based in reported and validated data and project history. The collaboration with the supplier (prestigious international company) continues in other scopes.
Complex Project distributed in two consecutive phases (Phase I – with a time constrained milestone due to the Euro introduction - and Phase II). Due to different causes the contractor performed only a great percentage of Phase I and a small percentage of Phase II. “Force majeuf” reasons made not viable for the contractor to finish the contract as adjudicated. It was decided a friendly contract resolution based in what really was implemented.
Exhibit 2 BCWP Calculations for Phase I
The Phase I of the Project was enough complicated that a main part of it, mainly new developments to solve “gaps” of the Standard features of the ERP in respect of the customer requirement specifications, that these gaps were converted in a new EVM subproject (LPGI) with priorities established by the customer sponsor and an emergency plan in parallel. Additionally, and in relation of Phase II of the initial project, had to be executed an important part of one of the modules composing this Phase II.
Resolution with EVM
Thanks to the utilisation of EVM, and to the restriction on the progress reporting indicated in the Technical Prescription Document it was possible to calculate the economic parameters of a friendly and objective contract resolution without major problems (agreement between the intervening parts) based on what was really earned. The Project circumstances, the simultaneous realization of activities belonging to the two consecutively initially planned phases, the conversion of part of the Phase I in a subproject with greater effort, priorities and an emergency plan can give an idea of how complicated the project had become. We have to add to this that the price of the project was fixed at the adjudication moment, with totally defined prices for both Phase I and II, and licences of the product used. Thanks to the EVM data (progress and effort invested), to the weekly project monitoring and to the tracking and control carried out we knew since early times the existing problems and could habilitate the corresponding emergency plan. The economic value based on what was really “earned” contributed to the “no objection” argumentation on the contractor side.
Phase I liquidation of the account
It was proceeded as shown in exhibit 2. At a certain moment the original planning of Phase I, had, after several revisions of the baseline, a finished and a non finished part which was decomposed in three major parts: Deliverables updates, LPGI (List of Pending Gaps and Issues), Emergency Plan.
Case 2 – Project Extra Cost Calculation
The Defence Organization. A International vendor of ERP. A consolidation and migration project with a schedule that span over a year. Around 40 participants one year project (between key users, IT Organization personnel, and the International Vendor Company). Due to special situation during the execution of the project and the Organization decision of stopping the project during a certain time due to operating conditioning, there was a cost overrun. With EVM we found a consensus way to solve the extra cost calculation.
To agree in the more objective way the cost incurred in a project due to different types of delays, some of which were induced by the customer (as he had to withdraw of the service, due to extraordinary reasons, the majority of his participants in the project team for two weeks)
It is a consolidation and migration ERP project, with more than one thousand “not standard” objects, migration from a support nearly discontinued version and with several possible scenarios to be defined, a lot of extra development and the existence of AddOns (components developed and maintained through a special agreement by the manufacturer, as it they were standard product until incorporated in a later release). The price of the project was close beforehand at a fixed price.
Resolution with EVM
It is summed up in the exhibit 3: EVM project situation.
Extra cost calculation and scenario change
To facilitate the calculations of the incurred extra costs, the entire project has been split into different parts, that were analysed separately in conjunction of the EVM input data and results.
- Part 1: Till the 22/10/2004 (The contractor receives a notification informing that the initially selected migration scenario had been rejected by the Organization due to internal reasons)
- Part 2: From the 22/10/204 to the 17/12/2004 (The contractor receives a notification informing the key customer staff allocated to the project had leave the project during one week due to special fiscal year closing)
- Part 3: From the17/12/2004 to the 21/01/2005 (Base Line review in order to estimate costs and impact and being able to chose a new migration scenario)
- Part 4. From the 21/01/2005 to the 01/04/2005 (the customer finally decides the desired migration date).
- Part 5. From the 01/04/2005 till the project closure
Exhibit 3. EVM project situation
Part 1 calculations:
On the 22/10/2004 it is indicated to the contractor that the initially planned migration scenario cannot be used. The contractor indicates that this decision will imply additional costs due to the migration scenario change. These extra costs were presented to the customer General Director for several possible scenario alternatives. The project situation on the 22/10/2004 were as follows (all data in man hours): BCWS = 9697, ACWP = 9744, BCWP = 8560, Schedule Variance = BCWS – BCWP =9697-9744=1137 mh, approximately equivalent to a project delay of 2,25 weeks. Cost Variance =ACWP-BCWP =9744-9697= 47 mh equivalent to 0,5%, deviation, that it was considered despicable.
According to the above data and was had being agreed in the execution contract, it was decided that any project delay till this date was not imputable to the customer. It was decided that the existing difference between ACBP y BCWP is only attributable to the contractor (1184 mh).
Part 2 calculations
Between 22/10/2004 up to 17/12/04, although the customer had rejected the planned January migration, it was stipulated that there was not any delay attributable to the customer, although formal protests exist on the part of contractor on the slowness of validations being made by the customer key users. The speed of validations, compared with previous validations carried out in previous projects with the same contractor were similar and accepted as normal, therefore it is not considered to be any delay attributable to the customer. The behaviour characteristics of the customer was something known to by the contractor during the phase of planning and it was indicated that he should consider this behaviour in the planning produce by the contractor (though agreed by the customer as the baseline).
Part 3 Calculations
In date 17/12/04 the customer key users stopped their project participation due to sudden priority change (urgent fiscal year closing issues). The delay originated was directly attributable to the customer. Although the rest of customer personnel continued working on the project it was considered as a simplification for calculating this project delay cost the work invested by the contractor during two weeks (the delay week and the following one, in order to take into account collateral and related effects of dependent tasks to be executed in following weeks) less the earned value obtained in these two weeks (effort in man hours). It was considered that two weeks was a good trade-off for taking into consideration de influence effects on related tasks on one hand and in the other the expected efficiency of the contractor in normal project managing endeavour.
Exhibit 4. EVM project situation
Table in Exhibit 4 presents the project situations at 17/12/04, 24/12/04 and 21/01/05 (expected deadline for an approved new project baseline).
It could be considered as delay imputable to the customer in this part 3, the contractor investment in the problem week an the following week (that is during these two weeks) less the earned value obtained during these two weeks. This means 769 man hour, that at the agreed mean project price was used to produce the agreed extra cost for this part of the project.
Part 4 Calculations.
The contractor indicates that is maintaining “artificially” s staff allocated to the project in order to “favour” the migration (not losing their staff due to allocation to another client projects). All the “Not completed tasks” of the project were analysed in detail as well as their dependence with the migration.
Exhibit 5 – Part 4 calculations
In order to validate this contractor statement we carry out the analysis sampling the tasks being executed in the project in three sample dates (21/01/05, 25/02/05 and 01/04/05). All the information form part of the reported data for EVM control. The result shows that after excluding those tasks planned relative the migration, the contractor personnel had been working in not completed tasks. This signifies to us that the contractor “had not maintained “artificially” personnel in the project, an therefore could not made claims on this respect, as there was always enough pending work, not directly related with the migration, to oblige the contractor to maintain his personnel “naturally” in the project. The result is shown in exhibit 5.
Part 5 Calculation
In this part there is a new baseline agreed between the intervening parts. We call familiarly this new baseline “replanning” We could distinguish in this “re-planning” three types of not completed tasks: 1. tasks remaining from the previous stage, 2. specific tasks related to the scenario change, 3. “forgotten tasks” (tasks that should be planned initially by the contractor and due to error were not included). From the extra cost calculation only those tasks relative to the scenario change were taken into consideration. The extra cost is calculated multiplying the agreed planned effort of the scenario change tasks in the baseline (1883 mh) times the average agreed cost of contractor personnel for the project.
Results, Conclusions, and Recommendations
The obvious benefits of the EVM are widely documented, and have been proved in the real life of the projects multitude of times. In this paper we have presented two uses or benefits of EVM, that the Organization had not thought or planned for initially. These not planned benefits, on their own justified the use of EVM, as the Organization:
- obtained substantial economic saving
- was able to base the deliberations on objective data, previously provided by the contractor.
- supported a stable relationship between contractor and customer
The most important recommendation of this experience is: Using EVM in fix price contract situations will help you not only in controlling and managing your project, but also will be a very valuable help in handling special contractor project situations not only if the project progresses appropriately, but also if it has problems in its execution
Solomon, P. (2002, October) Using CMMI to Improve Earned Value Management, Software Engineering Process Management, Technical Note CMU/SEI-2002-TN-016. Carnegie Mellon University.
Gansler, J. (1999,17 October)New ANSI Standard on EVMS Guidelines, ANSI/EIA-748-1998, Retrieved from http://www.acq.osd.mil/pm/newpolicy/indus/ansi_announce.html
PMI(2004) A Guide to the Project Management Body of Knowledge (PMBOK® Guide) Third Edition Newtown Square, PA: Project Management Institute
PMI(2005) Practice Standard for Earned Value Management Newtown Square, PA: Project Management Institute
Arnstein, D.M. (2004, Winter) The Seven Practices We Could (and Should) Implement to Improve IT Project Management, Absolute Consulting Group, Inc.
Christensen, D, S., Ph.D. (1998), The Cost and Benefits of the Earned Value Management Process. Retrieved from www.dau.mil/pubs/arq/98arq/chrisevm.pdf
©2006 Luis Cabezas Castillo
Originally published as a part of the 2006 PMI Global Congress Proceedings – Madrid, Spain