Are your projects on schedule? how do you know?

Introduction

Senior management desires to oversee projects within its organizational or business domain. Depending upon circumstances many independent or interdependent efforts may proceed in parallel under the direct supervision of project or task managers. Typically, upper management reviews these project efforts periodically. It is useful and efficient to gain a quick overall impression of the progress of a set of projects and focus on particular projects by exception. A convenient format is the color chart where the “health” of a project is characterized by a color. Usually, Green and Blue signify satisfactory and excellent performance while Yellow, Orange and Red depict increasing levels of difficulty. A Red or Yellow in a sea of Green quickly draws attention and interest to the project potentially in trouble. Supporting metrics then help to focus on the specific areas or tasks than need help.

All too often the status of a project is determined in a qualitative, sometimes ad-hoc fashion. The project manager might poll the project's team members to determine the percentage completion of particular tasks as well as the “percent-complete” of the project as a whole. These figures might be based on some quantitative analysis but frequently they are reflections of the subjective “feel” of the individuals close to the work. The values may also be computed automatically by a project management tool, such as MS Project. The problem now becomes one of ascribing a progress status (or “color”) to the effort, and that can only be satisfied by comparing actual progress against a baseline plan. Even then an algorithm must be employed to translate lagging (or leading) performance into a color status.

This paper addresses the desire for objective reporting in a consistent and repeatable manner. A technique and supporting tool, “MATS” (Measurement and Tracking System), are derived which exhibit the following characteristics:

• A standard, quantitative method of expressing a project plan in terms of “effort-components.”

• The a priori “effort-weighting” of such components to form the basis of a modified earned value criteria.

• The enforced discipline to recognize accomplished effort only when components are actually started or finished.

• The objective numerical comparison (or gap) between effort accomplished vs. effort planned.

• The application of heuristic algorithms, which derive color status, based on the derived gap and the stage of project expectation.

• The presentation to management of assessment data in terms of the ratio of accomplishment to plan, color status and graphical profiles.

The work described herein was motivated the necessity of reporting progress on a multiplicity of Y2K initiatives to the highest levels of organization management within a government agency. Some of the legacy programs were being renovated, other were being replaced. All efforts were structured along the lines of the GAO Y2K assessment guidelines. The technique and tool however have been generalized to apply to any IT circumstance where a number of parallel efforts are under way and management wants a quick and convenient way of monitoring and assessing progress.

The GAO Y2K Guidelines

The Government Accounting Office had formalized the necessary Y2K activities into five phases: Awareness, Assessment, Renovation, Validation and Implementation. From Reference 1 the Key Process Breakdowns are:

Awareness: Define the Year 2000 problem and gain executive level support and sponsorship. Establish Year 2000 program team and develop an overall strategy. Ensure that everyone in the organization is fully aware of the issue.

Assessment: Assess the Year 2000 impact on the enterprise. Identify core business areas and processes, inventory and analyze systems supporting the core business areas, and prioritize their conversion or replacement. Develop contingency plans to handle data exchange issues, lack of data, and bad data. Identify and secure the necessary resources.

Renovation: Convert, replace, or eliminate selected platforms, applications, databases, and utilities. Modify interfaces.

Validation: Test, verify, and validate converted or replaced platforms, applications, databases, and utilities. Test the performance, functionality, and integration of converted or replaced platforms, applications, databases, utilities, and interfaces in an operational environment.

Implementation: Implement converted or replaced platforms, applications, databases, utilities, and interfaces. Implement data exchange contingency plans, if necessary.

Key Processes further define each phase.

GAO Phases and MATS Components

MATS was applied to the GAO phases following Awareness. For the purpose of MATS usage the GAO Phase Key Process breakdowns were simplified and summarized into approximately six “components” within each phase. (The number of components is a design judgment. Executive oversight suggests an overall rather than detailed view of progress. Too many components introduce too much detail and data collection. Experience has shown that 6–12 components will suffice.)

Exhibit 1. GAO Weighted Components

Exhibit 1 lists the Assessment and Renovation components for two generic categories:

a. Y2K, meaning projects requiring Y2K remediation rather than replacement or new development.

b. Software Development, meaning projects involving new development in order to achieve Y2K compliance.

The table in Exhibit 1 also indicates “weights” for each component. These are preassigned measures of the anticipated effort contributed by each component within a phase. The total for each phase is 100%. The weights may be normalized with respect to budgeted hours, dollars or engineering judgment. MATS allows the components themselves and their relative weighs to be tailored for any particular project. In the case of software development for example, the GAO Phases customized and interpreted to fit any life cycle definition. The principle remains the same: establish a set of components that summarizes the key processes so that the effort can be monitored at an executive level.

The Approach in Using MATS

The Baseline

Any meaningful report, tracking and assessing progress must relate to a baseline plan, ideally expressed using a project management tool such as MS Project. For Y2K efforts, Project plans reflected tasks within the GAO defined phases, indicating required events, activities and milestones as a function of the calendar. Sets of tasks corresponding to MATS phase components were grouped and the earliest Start and latest Finish dates transferred to the MATS tool. The “earned value” weights were then ascribed to the components. The translation from project baseline tasks to MATS components requires some familiarity with the problem domain and some judgement in the grouping of tasks. The project staff itself, or an independent monitor, may designate these groupings by introducing special characters within the Project plan. Alternatively, MATS may be used as a stand-alone tool where the components are entered directly. For performance comparison across a similar set of projects, e.g., Y2K, the components were standardized. For each project phase (e.g., Y2K Remediation), a baseline plan was created. Each plan will reaches 100% accomplishment per phase according to a computed profile for the completion of its components; each component contributing a percentage increment until 100% is reached. The baseline plans may be expected to resemble, for example, the sketches for two of the GAO phases in Exhibit 2

Exhibit 2. Sketches of Progress vs. Plan

Note: For practical purposes it is assumed that planned activity and accomplishment of phase components occur in a linear fashion between component start and end dates. Actual accomplishment occurs discretely as the component is actually started or finished.

Using MATS

The process of using MATS for measuring and tracking progress is straightforward. The MATS Excel tool contains an input sheet for each reporting period, and a Summary Sheet. Initially the project baseline, i.e., the planned start and finish dates for each phase component is transferred from MS Project or entered directly on the Enter Plan-Sheet. Expected dates (if available) may also be entered for the entire project, as “temporary actuals,” on this sheet as well in order to view anticipated performance. With greater knowledge and experience these might be different that the baseline plan. As the project progresses actual component start and finish dates are entered on succeeding reporting sheets. The Summary Sheet shows the computed results, i.e., an evaluation for each reporting date, graphs for each project phase showing actuals vs. plan and an evaluated color assessment for each phase, viz. Blue, Green, Yellow, Orange or Red. Evaluation is a function of the comparison between value actually earned in any period vs. value planned to be accomplished in the baseline. Note that while MATS accommodates multiple project phases, a project may also consist of a single phase.

Metrics

The assessment of the progress of a project depends upon a comparison between “value” actually earned by accomplishment vs. value planned to be accomplished in the baseline. In terms of earned value this is a comparison between the “Budgeted Cost of Work Performed” vs. “the “Budgeted Cost of Work Scheduled,” i.e., BCWP vs. BCWS. For Y2K actual cost performance (ACWP) was not observed. (In many organizations cost collection by task or component is problematic. However, MATS is now being modified to account for collected actual project costs.) In this sense, MATS is a modified earned value system. Each component is given an effort-weight thereby producing a planned effort-profile vs. calendar time for a project. As the project progresses portions of the budgeted weights are “awarded” according to accomplishment. The cumulative award is then compared to the baseline profile and a reporting assessment color is determined. The quantitative comparison is based upon the Schedule Performance Index (SPI), viz.

SPI = BCWP/BCWS,

or in MATS terms,

= % Effort Accomplished / % Effort Planned for each phase.

The Effort is computed from the sum of the efforts of each component at any reporting calendar time. The color assessment is arrived by applying the following heuristic criteria:

Accumulation of Baseline Value

A simplification, and a reasonable assumption is to assumed that between planned baseline start and finish dates, for any component, accumulation of effort is linear with time, i.e., a straight line. The total baseline profile, over calendar time, therefore is the summation of the straight-line profiles for all the components.

Accumulation of Actual Value

Actual accumulation is more problematic. In order to avoid subjective estimation, e.g., a guess by project staff at “% complete” of any task or component, a common approach is to award half the effort when a component actually starts and half when it actually finishes. Since a component is a collection of project tasks, 50% for the component is therefore awarded when any constituent task begins and the remaining 50% awarded when the last task of the set finishes. This is effective but does exhibit granularity in initially being optimistic and then becoming pessimistic. The larger the large number of components within any phase, the more this effect is smoothed because the effort ascribed to any one component is correspondingly smaller. This “granularity” effect is acknowledged and accounted for in the final evaluation. The choice of 6–12 components within any phase is a compromise between granularity and the desire to minimize the amount of reporting data.

Granularity Effect

Granularity is caused by the discrete nature of the weightings assigned to components. If a component were assigned a weight of, say 30%, then when it actually starts 15% would be awarded; the remaining 15% is awarded when the component actually finished. At the very beginning then, the actual leaps 15% ahead of the plan; near the finish it may be 15% behind the plan. Even if the project were proceeding exactly as planned, the error could be as large as 15%, i.e., 1/2 the assigned weight, or one quantum. This granularity affects the color determination. The consequences are treated differently depending upon the baseline BCSW expectation. The heuristic chosen has the following levels and operations:

a. When the baseline BCSW calls for accomplishment <25%, i.e., when a phase is just beginning, evaluation can afford to be liberal (there's along way to go) and granularity can be very confusing. The algorithm is to view the project as Green if any actual accumulation occurs. For example, if the baseline calls for 24% and the actual accumulation is only 5%, the project assessment is Green. However, if the actual is 0%, i.e., no recorded progress at all, the rating is Red. This logic tends to give the benefit of the doubt to progress at the beginning, particularly because of the granularity and the uncertainty. This approach may be overly optimistic, at the start of a project, but it is difficult to quantify a project with no track record. At any rate, without a vigorous uptick in actual performance, thereafter, the color assessment will soon turn to Yellow or worse.

b. When the baseline calls for accomplishment between 25% and 50%, and the difference between the actual accumulation and the baseline is within one “quantum,” i.e., 1/2 the maximum component effort within a phase, the project is Green. Outside of one quantum, the Metric Value and color are computed from the previous table. This logic recognizes the granularity inherent in the measurement and allows for a Green assessment within one quantum of the plan.

c. When the baseline calls for accomplishment greater than 50% up to 70%, and the difference between the actual accumulation and the baseline is within two-thirds quantum the project is Green. Outside of two-thirds quantum, the Metric Value and color are computed from the previous table. Note that the logic here enforces a stricter criterion for Green as the project baseline expects higher achievement.

d. This tightening is continued in demanding more performance as the project heads into the home stretch. When the baseline calls for accomplishment greater than 70% up to 90%, and the difference between the actual accumulation and the baseline is within one half quantum the project is Green. Outside of one half quantum, the Metric Value and color are computed from the previous table. Note that the logic here enforces a stricter criterion for Green as the project baseline expects higher achievement.

e. If above 90% and the difference between the actual accumulation and the baseline is within the minimum of (10% or one half quantum), the project is Green. Outside of this criterion the Metric Value and color are computed from the previous table. Again this tightens the criteria further as the project baseline approaches 100%.

While the algorithms above constitute heuristics they have been applied to a number of projects with intuitively satisfactory results. The formulas are mechanized within the MATS tool and the resulting colors do convey the status of the projects. The Yellows, Oranges and Reds point to marginal and serious problems needing management attention.

Using MATS

The use of MATS is illustrated by applying it to two Y2K initiatives. Some of the data has been altered to illustrate features of the MATS tool. The steps followed in establishing the MATS models are:

• Step 1. The MS Project plans for each initiative were first analyzed and work package tasks were grouped into the phase components.

• Step 2. % Weights were applied to each component within the phases using the percentages shown in Exhibit 1 as a guide. (Weights may be tailored to suit a particular project.)

• Step 3. The planned Start and Finish dates for each component were transferred from Project into MATS onto the Enter-Plan Sheet in MATS. The data for Project A is shown in Exhibit 3. Note the component descriptions and the % Weights.

• Step 4. The actual dates (i.e., dates components actually started of finished), or “expected” dates, were also entered. The reporting dates, i.e., the dates on which a report is delivered to executive management, are also inputs to MATS and for illustrative purposes have been initially set at the 24th of each month. The reporting date is easily changed to reflect the actual executive review. All “actual” dates prior to the review date are considered as actual accomplishment; dates subsequent to review are viewed as anticipated.

Exhibit 3. Project A MATS “Enter-Plan” Sheet

Exhibit 4. Project A Renovation Summary Sheet

Exhibit 5. Project A Renovation Profile

• Step 5. All plan and actual dates are displayed on the Project A Renovation MATS Summary Sheet, Exhibit 4. Corresponding to each reporting date, the planned percentage achievement for the phase is shown along with the actual or expected achievement (%) for that date as well. Note that “actual achievement” corresponds to objectively computed “% complete.” Using the quantum value for granularity, as explained above, the result is computationally adjusted so as to yield the correct Metric Value (“Adjusted Ratio”) and the assessment color for each reporting date.

• Step 6. MATS also produces graphs for each phase. Exhibit 5 shows the results for the Renovation Phase of Project A. The baseline plans as well as the actual or expected achievements for each of the phases. As the project progresses, actual dates and specific reporting dates are entered into the MATS reporting sheets. The Summary Sheet and the derived graphs always show the current status, the history and the anticipated performance.

Summary

This paper presents a monitoring approach and tools that are suitable for executive management oversight of multiple projects. They are based on a modified earned value concept and encourage objective consistent measurement and reporting. The MATS tool provides the mechanisms for addressing project plans, reducing them to standard criteria and comparing actual vs. planned accomplishment. Analysis enables the pinpointing of problem projects as well as problem task areas within those projects. The monitoring data is conveniently packaged in tables, graphs and finally, “fever chart” colors for ready and comparative assessment. The approach and the tool have been in active use on a large, mission-critical government Y2K effort. The suggested method and MATS tool may be applied as a tracking and reporting mechanism for most software development and IT projects. A single phase or multiple phase life cycle may be utilized. Because the method embraces standard structures it may be particularly useful for executive management reporting where a number of simultaneous projects require oversight.