The truth about preparedness for any EVMS assessment by others
This paper focuses on lessons learned and best practices in order to reach a high state of readiness for a serious “audit” of your Earned Value Management System (EVMS) design and use. Excellent preparedness is essential if we realistically expect the results to be successful. To achieve the best level of readiness we must understand the principal factors which contribute to our preparedness and how to overcome both foreseen and unforeseen obstacles. Many typical challenges and both common and uncommon solutions to readiness will be addressed in the presentation. Much is at stake—ANSI compliance, pride, market reputation, and bragging rights!
Most serious reviews by external assessment teams of your EVMS are driven by the need to demonstrate compliance with American National Standards Institute (ANSI)/Electronic Industries Alliance (EIA)-748-B, “Earned Value Management Systems.” Most often referred to as EVMS Certification or Validation Reviews, such reviews are conducted to evaluate three aspects of the EVMS: it’s definition and design, customarily documented in the contractor’s system description and procedures; the fullest of its implementation and use by the program/project teams; and, the availability and use of timely and reliable performance measurement data by all relevant program/project stakeholders.
In addition to certification/validation reviews, often major Surveillance Reviews are also conducted, by the customer or its representative, to determine whether the previously validated system remains compliance with contract and ANSI 748 requirements.
However, a third type of review may also be experienced in which many aspects of the EVMS are also reviewed by a joint contractor and/or customer team. This is the Integrated Baseline Reviews or the Technical Independent Project Review. The former is applicable to U.S. Department of Defense contracts, whereas the latter is applicable to Department of Energy contracts and projects. Some U.S. government civilian agencies also conduct similar reviews under various names. Most such reviews typically focus upon the quality of the scope, schedule, and cost related planning of the target program/project to assure the project is fully planned and achievable, including identification of risks and planned, appropriate risk responses, and that adequate monitoring is possible through routine, reliable performance, measurement data.
Exploration of the principal elements, which impact a company’s readiness and evidential success when confronted with one or more of these reviews, is addressed in the remainder of this paper.
Principal Elements That Impact Assessment Review Preparedness
Though many contributing factors can affect the readiness and successful results regarding a major review of our EVMS, they can be grouped into tangible and intangible categories. The tangible elements include the following:
- information systems, consisting of hardware and software and often simply referred to as the “tools set”;
- management processes or process groups; and,
- artifacts and performance measurement data, often simply referred to a “EV metrics.”
Intangible elements include the following:
- people skills and talents;
- people attitudes and behavior
- organizational culture
It is assumed here that the tangible elements of the EVMS are fundamentally in place and functioning and that the project team and other EVMS stakeholders have demonstrated an adequate understanding and capability in applying the intangible elements. Said another way—an EVMS has been designed and implemented on the target project.
Now the challenge is to be very well prepared when an external authority arrives on-site at the contractor’s facilities to assess the quality and role of the EVMS in managing the project.
How to Assure Preparedness of the Principal Elements
When readying for an external assessment of your EVMS, each of the principal elements that impacts the success of the assessment must be the focus of a thorough and thoughtful review and appropriate actions to prepare both the tangible and intangible elements of the EVM System. The preparedness effort should be approached by planning it out, assigning roles and responsibilities, and tracking progress toward closure of each area of preparedness involved. The effort should lead by a senior management sponsor or responsible directorate.
Preparations for an external assessment require special attention beyond the ordinary day-to-day project management routine. It often is a shared, joint effort between the target project management team and another functional department (e.g., a PMO, Finance, Quality Assurance, or Internal Audit organization).
The information system is usually a combination of legacy systems and project specific software, usually schedule and/or cost processors (e.g., Primavera P3/P6®, MS Project®, Cobra®, MPM®, etc.). Sometimes all components of the are integral parts of an Enterprise Management System (e.g., SAP, Oracle), but not often. In any case, the EVMS tool-set components and the overall system architecture are established long before special attention is given to preparations for an external assessment of the EVMS. The implication of having an already established EVMS-related information system is that minimal, if any, ‘readiness actions’, which may be carried out on the other elements impacting preparedness, would be available to the Preparations Team 1 to 3 months prior to the actual Assessment review.
Most software systems set-up, integration, and overall architecture define the functionality and practices of the actual EVMS tool-set in place during an assessment review. Therefore, unless a failure or omission of needed functionality is discovered in the period shortly before the actual Assessment Review, no substantive attempt to modify the tool-set for preparedness purposes is practical in the window of time normally available for assessment review preparations. Hopefully most, if not all, tool-set issues are identified and resolved during the several months of routine monthly data cycles prior to the period when assessment review preparedness is being carried out. However, knowing information system-related issues typically cannot be easily rectified just prior to an assessment review, it becomes clear that there is a need to carefully design and set-up the entire EVMS tool-set, including legacy systems connectivity, during the initial EVMS development and implementation phases. The early System operational phase should focus on debugging and validating the system.
What can be carried out in the preparation period is an internal audit of data traces through the total EVMS tool-set. This should include a careful review of the integration of cost and schedule planning, schedule vertical and horizontal integrity, and labor, material and subcontract cost integration with other performance measurement metrics (e.g., Budgeted Cost for Work Scheduled and Budgeted Cost for Work Performed). The correct identification and recording of accrued costs ( also known as “estimated costs”) are especially important elements to assure the tool-set handles correctly. Often too, EVM Systems design requires consideration for some manual input of accruals to assure costs are correctly and completely recorded. This is because automated system accruals may not capture all the needed accruals. When a combination of automated and manual input of accruals are involved, it assessment review preparedness must focus on both the information system operational readiness and correct execution of the data accumulation process.
The data traces mentioned here are also applicable to the tangible element of performance measurement data, which is addressed more specifically later.
The Defense Contract Management Agency DCMA, executive organization with the U.S. DoD identifies nine process groups, which fit within, or contribute to, ANSI 748 compliant EVM Systems. The process groups identified by the DCMA are:
- Budgeting/Work Authorization;
- Indirect (cost) Management;
- Management Analysis;
- Change Incorporation;
- Material Management;
- Subcontract Management.
In practice, it is rather common to redefine or re-categorize these processes to fit company practices; however, their collective necessary functionality in meeting EVMS requirements must be maintained.
It is recommended that when developing or improving processes of the EVMS, the processes are documented in something such as Visio® software, using the multi-functional template. This greatly improves the clarity and integration of the processes.
Preparations regarding EVMS-related processes usually focus on conducting thorough training to assure the project manager, control account managers (CAM), project control staff and other stakeholders understand the processes and implement them correctly. In addition to training, however, a careful review of the effectiveness and integration of the processes of the EVMS should be carried out.
Assuring processes are fully implemented is an important readiness step, which can be facilitated by developing a live-document flow diagram (also known as a “storyboard”) for each relevant process. The storyboard should be built from live EVMS artifacts implemented and in place on the target project. Expanding the sample size to multiples may be wise if there is concern that full, correct implementation of the process has not been achieved.
We are reminded by the work of Dr. W. E. Deming, the father of modern quality management, in which he demonstrated that the vast majority of quality problems are rooted in processes—ignored, flawed, or misunderstood by the worker, not in the workers themselves. Clear process definition, documentation, and accompanying roles and responsibilities are vital to assuring that EVMS works as intended and compliant with ANSI Standard 748.
Artifacts and Performance Measurement Data
The reliability and timeliness of performance measurement data are of paramount significance to assuring maximum readiness for any outside sponsored assessment review. The basic eight performance measurement data elements from which all other metrics are derived are:
- Budgeted Cost for Work Scheduled (BCWS), as known as “planned value”;
- Budgeted Cost for Work Performed (BCWP), as known as “earned value”;
- Actual Cost of Work Performed )ACWP), also known as “actual cost”;
- Schedule Variance (SV)
- Cost Variance (CV)
- Budget at Completion (BAC)
- Estimate at Completion (EAC)
- Variance at Completion (VAC)
Those involved in carrying out actions of preparedness regarding the data must keep in mind that experience teaches us that the unwritten rule of government validation review teams is that processes or practices that distort, or tend to distort, performance measurement data are considered non-compliant with ANSI Standard 748. Therefore, preparations for an external assessment review must scrub the data, identifying errors and omissions, and rectifying them as soon as possible. Do not forget Deming’s teachings—be sure to look for process-related weaknesses, not just correction of the data.
Artifacts, or documents, forms, templates, and reports, are objective evidence of the EVMS existence, and as such, much also be prepared for an Assessment review. The typical implemented EVMS consists of many artifacts, usually including:
1. Work Breakdown Structure
2. Organizational breakdown Structure
3. Responsibility Assignment Matrix
4. Work authorizations documents
5. Budget spreads (time-phased)
7. Control Account Plans
8. Internal and external Performance Measurement Reports
9. Variance Analysis Reports
10. Baseline Logs
11. Baseline Change Requests
12. EVM System Description and procedures
Care must be taken to assure that these artifacts are designed and implemented consist with EVMS requirements. Typically, continuous monitoring of EVMS operations identify quality issues, but during the Assessment Review Preparedness window in the weeks before the Review, these documents should be sampled in all areas of potential weakness to assure they meet documented processes and procedures and EVMS requirements.
Specifically for the EVM System Description, a cross-reference matrix mapping the sections of the Description to the 32 specific Guidelines of the ANSI Standard 748 should be prepared if it doesn’t yet exist. It not only helps the Assessment Team’s review process, it also serves to assure that all relevant requirements are addressed through processes and artifacts of the implemented EVMS.
The traceability tests referred to in the Information System section earlier are also applicable here. Data traces should be performed for work authorization (scope definitions), budgets and schedules by tracking their flow from each artifact to each artifact, and from top to bottom of the System. Likewise, cost records should be traced from original source documents to final ACWP reports This includes the accrued costs as well as posted actuals in the legacy finance records. It is wise to perform the traces with several set of ‘live’ data to maximize the reliability of the traceability results.
Data traceability is an EVMS requirement especially for budgets, as it is explicitly cited by one of the 32 ANSI Standard 748 Guidelines. Special attention should be given to assuring the budget traceability, from top to bottom of the system and for all baseline changes are identified and documented in a controlled manner, at least at the control account level. Typically, it is expected that this requirement is fulfilled by the establishment and maintenance of a Master Budget Log, or equivalent.
People Skills and Talents
Maximum preparedness regarding the skills and talents of the relevant staff originates with the selection of qualified and motivated people to fill EVMS roles, especially the project manager, control account managers, and project control staff positions. However, that is not enough. The EVMS development and implementation team should include the System users; i.e., processes development or improvements should be reviewed with users; planning should be performed by responsible CAMs; and reviewed and approved by responsible management. The benefit of involving the primary EVMS users in the development and implementation of the System is stated well by an ancient Chinese saying:
“When I hear…I know
When I see…I believe
When I do…I understand”
The “doing” stage of the involvement of EVMS users must be carefully managed, first by preparing and presenting sound training based upon well-documented EVMS processes and key system artifacts (e.g., work authorization documents, performance reports, variance analysis reports, baseline change requests, etc.). Implementation typically takes a minimum of 2 or 3 months, followed by routine monthly progress statusing, analyses, changes control action, and forecasting estimates to completions. These actions, principally carried out by CAMs, need to be repeated month after month for typically 6 or more months in order to develop the necessary competency before an outside assessment review team arrives on-site.
Near-term preparations for the assessment review can rely upon the practice period to prepare the staff for the review, but additional refresher training should be provided to all system users. The topics for refresher training should be selected based upon the months of monitoring and mentoring the staff implementation and system usage. Subjects such as statusing of schedules, accruing costs for materials and subcontracts, updating estimates to complete, and possibly others should be established as separate workshops, using a hands-on approach to reviewing live examples of the documents and data to understand where weaknesses exist and how to improve practices and data. Such workshops are often essential to flush out the problem areas and help the staff make meaningful advances in the competency in EVMS understanding and usage.
The know-how of CAMs in using the system and demonstrating competency during assessment review team interviews is critical to the success of any such assessment. Assuring maximum preparedness involving people skills should include individual one-on-one coaching of key EVMS role occupants (e.g., project manager, CAMs and project control staff members). Coaching should be carried out by subject matter experts with experience in EVMS implementation and assessment reviews. Though qualified internal resources for coaching are desirable, they may be unavailable. In such cases, outside expertise should be sought.
A sound preparedness plan will also include performing mock interviews of project team staff – every CAM and the project manager. The interviews should typically last 2 to 3 hours and simulate actual Assessment Review Team interviews. It is wise to provide the opportunity for at least mock interviews for each interviewee within the 8 to 10 weeks prior to the actual assessment review.
Attitudes and Behavior
Like some other elements of preparedness, the origins of attitude and behavior of the project staff start with the careful selection of the project team, and the EVMS Implementation team. Well chosen individuals go a long way toward assuring the right attitudes among the project staff, however, as we know, sometimes the best people are not available or other factors interfere with our ability to make the best choices.
The single most important element to assuring that constructive, motivated and empowered people make up the project team and supporting staff, and therefore also the EVMS users, is the leadership of a company’s senior management. Experience teaches us that leadership is vital to an efficient and successful implementation. The fundamental message, which management must convene is word and deed is that EVMS is important, the only such system to be used to manage the project, and that everyone is expected to understand and use it in managing their project work.
A common means of assuring readiness of all aspects of the EVMS—tools, processes, people, and data, both the tangible and intangible elements of the System—is the conduct of a Mock Assessment Review by an independent third party; one experienced in EVMS implementations and the details of external Assessment Reviews. Such mock reviews simulate all aspects of the real Assessment and provide a clear baseline of the readiness of the EVMS and target project team to undergo an external Assessment Review, including determining compliance with ANSI Standard 748.
Organizational culture can be an asset to effectively implementing and using EVMS to manage the project work or it can be a significant obstacle. Culture consists of the collective style, attitudes, and expectations of an organization or group, molded, maintained or changed only by the organization’s senior management. Like staff attitudes, leadership by senior management sets the culture, and is the only means by which the culture can be changed.
Preparedness regarding culture cannot be assured in the weeks prior to an externally sponsored assessment review; it is built from within over a longer period of time and is most often slow to change. It is best to recognize the cultural element in developing and implementing any management system, including EVMS. Taking notice of cultural assets and harnessing them to aid in the EVMS development, implementation, and use from the outset are important. Likewise, cultural barriers should be recognized for what they are, and EVMS implementation team leadership should work with senior management to help overcome them. These actions must also be carried out from the outset.
Lastly, a focused effort by a senior management sponsor or champion for EVMS Assessment Review preparedness should be evident to all, providing leadership, clarifying management expectations, and assuring accountability among all concerned to fulfill all actions of the Preparedness Plan for the upcoming Assessment review.
Coordination with the Assessment Review Authority
Several aspects of coordination with the Assessment Review Authority must be attended to in preparation for the on-site Review. Such actions are carried out as required steps leading to the Review, and thus represent minimum preparedness actions. Regardless of how thoroughly prepared the contractor wants to be, such coordinations are required. The specific coordination steps vary from sponsoring agency to agency, but fundamentally involve some type of readiness assessment to confirm the contractor’s preparedness for a full Assessment review. As the date established for the Review nears, the contractor is expected to transmit an advanced data call of information to the Review Authority. The data call is typically due to the external authority 30 days in advance of the on-site Review.
The data call often requires extensive documentation to be forwarded to the Review team for its inspection prior to arriving on-site. Common items in the advanced data call include the following.
Earned Value Management System Documentation and Others
- EVMS System Description
- EVMS-related Procedures and others
▪ Work Authorization
▪ Data Accumulation, Reporting, Analysis
▪ Estimate to Complete/Estimate at Completion
▪ Change Management
▪ Material & Subcontract Management
▪ Indirect Cost Management
▪ Risk Management (related, but not EVMS specific)
▪ Corporate Business Systems Architecture
▪ Others (TBD)
- EVMS Storyboard (if available in 11x17 format)
- Contractor Contract clauses relevant to EVMS requirements & reporting
- Contact List of all Contractor direct EVMS responsible staff, including primary roles in EVMS and Validation Review support)
Project Specific Documentation
- Customer contract authorizations of project
- Project Execution Plan (current version)
- Project WBS
- Project OBS
- Project RAM dollarized (including contact list of all CAMs and PM)
- All Work Authorization documents (Preliminary and Baseline, if applicable)
- Project Schedules (all levels, including most detailed version)
▪ Baseline Schedule
▪ Statused schedule (for each of prior 3 months)
▪ Schedule Critical Path (for each of prior 3 months)
▪ Overview of Project Major Milestones (in whatever form they exist)
- Control Account Plans (all)
- Internal Performance Measurement reports at all level of summation (prior 3 months)
- External Performance Measurement reports for customer (CPR or equivalent) (prior 3 months)
- Variance Analysis Reports (prior 3 months)
- Project Master Budget Log
Simply designing and implementing an EVMS are not sufficient to assure full readiness for an external team’s major assessment review of all or part of the EVMS. The Earned Value Management System involves no less than eight or nine major management processes, which must work together to provide integrated planning and control of project scope, schedule, and cost parameters. Systems, processes, and people must all work together to operate an EVMS successfully. Preparedness of the EVMS, its toolset, processes, artifacts and data, and people requires substantial planning and attention to details. To achieve this and be fully prepared for a successful Assessment Review requires special focus on all elements impacting both the readiness and success of the upcoming Review. These elements include six primary area of readiness:
- Information System
- Management Processes
- Artifacts and Performance Measurement Data
- People Skills and talents
- Attitudes and Behavior
- Organizational Culture
A comprehensive Preparedness Plan should be drafted, responsibilities assigned, and progress tracked frequently to assure that all matters are being attended to, issues are understood and dealt with, and resources and necessary cooperation are available throughout the organization. A Mock Assessment Review is also an effective measure of preparedness, often helping to baseline the degree and content of Preparedness Plan actions needed in the weeks prior to the real Assessment Review. When maximum readiness is sought using these sound management disciplines and support, led by a committed member of the senior management staff, the probability of success is often greatly enhanced.
©Richard E. Brodkorb
Originally published as a part of 2011 North American PMI Congress – Dallas/Ft. Worth, USA