Abstract
Earned value (EV) measurement and techniques, as methods for project management monitoring, reporting, forecasting, and controlling have been developed and adopted over the past few decades in software product development as well as traditional engineering projects. The methods, relying largely on baselined task-driven plans with fixed and well-defined scope, assume linear progression on task execution and completion that pave the way to the devising of schedule and cost performance indices that can be used for forecasting and making decisions on project controls. While Agile Manifesto does not mention the words “measurements” and/or “metrics,” and the approaches do not readily lend themselves to return on investment (ROI) metrics, the authors identified the need to complement their projects’ performance monitoring, forecasting, and control techniques by adding simplified earned value management (EVM) methods to the burn charts and velocity calculations. Short sprint cycles are helpful, and project risks and uncertainties get reduced by delivering regularly and quickly; but, while the product backlogs remain dynamic, as a result of sprint spillovers and stakeholders’ fine-tuning of features and requirements, the additional performance tools provided by EVM are still required to communicate the “bigger picture” of end-to-end project status and forecasts to the non-agile stakeholders, as well as for input to the project control and executive decision-making process.
Introduction
The Agile method used for this study was based on Scrum. Because in Scrum the work scope and the performance requirements are fixed for sprints (barring any major issues) the tracking of status and performance monitoring is not a big issue for the short cycles. Because the items in product backlog haven’t been committed to for future sprints, then there is every possibility they will change, and the burn charts focused only on the current sprint, provide little real insight to possible future performance, not to mention the project end-state metrics. Indeed, the more Agile the team and the project environment is the more likely the work and the requirements are to change. However, objective end-to-end status and performance monitoring is still needed to enable tracking and trending assessment, and to forecast possible cost and schedule variances from the planned or desired project end points.
While Agile techniques can provide for measuring “velocity” and predictability of the current iteration, their use to forecast future project performance or project end-state metrics is marginal at best. The authors of this paper will present their approach to performance measurement of agile-run projects, to narrow the gap between total agility and legacy conventional techniques. We will show how adopting a simplified EV approach can help determine an Agile project’s true status, detecting early signs of trouble, as well as providing end-state forecasts based on up-to-date project data.
Agile Projects and EVM
While the roots of EVM and earned value analysis (EVA) stretch all the way back to the late 19th century Industrialists and the rise of industrial engineering, EVM and EVA were finally developed into a standards-based process and methodology based on mathematical formulas during the 1960s. A driving force behind the development was the Defense and Energy Departments who, at the time, were funding large, complex, expensive, risky projects ranging from aircraft carrier weapons systems to nuclear reactors. The key benefit of using EVM/EVA for these projects was that the mathematical combination of cost/spend and technical progress measurement metrics gave project managers, and thus stakeholders, a set of standard, forward-looking metrics on which to make timely decisions.
EVM methods stress the importance of establishing a measurement baseline before the project starts. From this baseline of time and budgetary values, actual hours, costs, and technical progress are compared for the duration of the project. This comparison can be made at the task level, an intermediate “roll-up” level (the control account) or at the total project level, thus providing a standards-based multi-level controlling and reporting solution.
As the role of enterprise software began its relentless growth as the competitive edge for large companies in most industries, software projects in the 1980s began to take on the character, cost, and risk of the weapons systems and nuclear reactor projects of the 1960s. Large government software projects were required by their sheer cost, risk and complexity to implement EVM. However, adoption of EVM by the private sector for large software projects was slow until the Federal Government was able to pass the EVM baton to the private sector with the creation of ANSI/EIA-748 Standard, in 1992 (Athey, 2007).
Large software projects of the pre-Internet era exhibited characteristics similar to the 1960s weapon system and nuclear projects. With the pace of technology slower than today, it was possible to plan these software projects against a static requirements baseline with a valid lifespan equal to the duration of the project. As enterprise software grew into a “weapon” of economic competition, the ability for project planners to establish a requirements baseline and manage to it for the life of the project rapidly diminished.
Planning baselines and functional requirements had to gain flexibility. As the need for baseline flexibility and inprocess modifications accelerated during the 1990s, the prevailing view became that EVM was too cumbersome and costly for most private-sector software projects. The end of the waterfall, Gantt-based-long-term-planning-static-baseline software projects was rapidly approaching, coincidentally as the EVM standard passed from Government to industry in 1992, as ANSI/EIA-748 (Athey, 2007).
By 2000, a number of Agile software development methodologies and frameworks, were documented and in use. In February, 2001, 17 people representing the most prominent Agile methodologies gathered to find common ground across their similar but differing approaches to agile software development (Highsmith, 2001). eXtreme Programming (XP), SCRUM, Dynamic Systems Development Method (DSDM), Adaptive Software Development, Crystal, Feature-Driven Development, Pragmatic Programming were all represented at the conference, which formed the genesis of an alliance to foster the growth of Agile. The conference attendees agreed to articulate their common principles in the Agile Manifesto (Highsmith, 2001).
Soon after the Agile developers’ gathering in Utah, Ken Schwaber and Mike Beedle (2001) published their landmark text entitled: Agile Software Development using Scrum, describing how to implement XP quickly and easily by cutting through the complexity and ambiguity of emergent user requirements and unstable technology, to iteratively produce quality software. Many believe that SCRUM is the fastest, easiest, and most widely accepted Agile process.
EVM, Re-thought and Applied to Agile Methods
As developers adopted Agile methods for software projects large and small, many thought that the new era of Agile was going to be the end of EVM’s use for software projects. Arguments arose that EVM wouldn’t work without a work breakdown structure (WBS) and a baseline of fixed requirements and, that notwithstanding, EVM is at best, cumbersome in the midst of rapidly changing requirements. Additionally, most Agile methods decompose a project into a series of iterations, that for all practical purposes, are themselves separate projects. If the baseline values are unrealistic, requirements change frequently, and each iteration is a (say) four-week project, then EVM can’t work and its application to Agile projects is a waste of time, the argument went (Ahmed, 2005).
However, a search of the literature reveals a number of articles that began appearing soon after the emergence of the Agile Manifesto, advocating for application of EVM to Agile projects. A central theme of these articles was that EVM will have to adopt new terminology and adapt its definitional metrics to be meaningful in an Agile setting, and that there are convenient analogies that can be constructed between the metrics of EVM and the metrics of Agile.
In a February, 2003 article in PMForum Glen Alleman described a method for applying earned value techniques to iterative projects. In the article, he advocated using the 0/100 EV method on “testable requirements” completion. Then, Messrs. Alleman, Henderson and Seggleke, presented a paper at the June 2003 Agile Conference that describes what was required to introduce Agile methods into a “high-ceremony” government contracting system, which required EVM as the project performance reporting and management method (Alleman & Henderson, 2003).
Alleman, Henderson, and Seggleke found that applying the 0/100 EV method to measure completion of XP-based project Stories, Tasks, and Testable Requirements yields results that look strikingly similar to an Agile software development process. “With this fine-grained task breakdown process, all the EVMS principles are still in place, but the behavior of the management system has many of the attributes of an agile process” (Alleman & Henderson, 2003). The writers demonstrated that it is possible to create a software development environment that adheres to many of the XP practices while maintaining ANSI/EIA-748 reporting compliance by establishing the following analogies (Alleman & Henderson, 2003):
- Replacing XP’s velocity with EV metrics
- Creating fine-grained measures of budgeted cost of work performed (BCWP) using “testable requirements”
- Establishing the Budgeted Cost of Work Scheduled (BCWS) baseline at the beginning of each iteration
- Capturing Actual Cost of Work Performed (ACWP) through a time keeping system
- Computing Cost Variance, Schedule Variance from the three base earned value metrics
- Computing Estimate at Completion (EAC) and Estimate to Completion (ETC) from these base metrics as well.
One of the principal Agile performance tracking techniques is based on “burn charts,” which correspond well to the measurement concepts of EVM. Burn-up and burn-down charts are both used in Agile methods, the former to show the increasing amount of functionality completed by the project team over time and the latter to show how many features remain to be completed. In either chart, only running, tested features can be counted for the purpose of displaying either the number completed (burn-up chart) or the number remaining (burn-down chart.)
Alistair Cockburn (2004), one of the Agile Manifesto co-authors, published a detailed treatise on his website entitled “Earned-Value and Burn Charts,” comprised of information extracted from his book Crystal Clear. Cockburn’s premise is that Agile “burn charts” have a ”…natural mapping to the earned value charts used in military/government projects. They should be part of your standard bag of tricks for project planning and reporting” (Cockburn).
The Agile burn-up chart closely resembles the EV and PV plots of the classical EVM 3-line graph of cumulative EV, AC, and PV. In Figure 1, for a SCRUM project, the top line shows planned feature completion (analogous to PV) over the five iterations of the project, and the lower line shows actual feature completions (analogous to EV) through Iteration #4.
As seen in Figure 1, at the completion of Iteration #4, the project is approximately 67% complete (67 feature pts. / 100 feature pts.) and it currently has a negative schedule variance (SV) of approximately 9% (67 feature pts. completed – 76 feature pts. planned). Knowing that each iteration in this example project is four week’s duration, the 9% negative SV can be converted to approximately two weeks behind planned schedule (9% of 20 weeks total).
Figure 1. Sample Burn-Up Chart Showing SV
All that is missing in this example is a plot of AC; however, burn charts have typically not tracked cost for a variety of reasons, not the least of which is that they are maintained by the development team instead of the project manager. Cost tracking and cost variance analysis have typically not been the domain or concern of Agile development teams. Cost variance, also a key EVM metric, is difficult to measure or even infer using typical Agile metrics. Consequently, while most Agile methods embrace the need for recognizing stakeholder ROI, they offer little guidance for measuring and reporting ROI or its primary component, project, or iteration cost.
SV, on the other hand, can be easily measured using Agile metrics, as shown in Figure 1. When schedule variance occurs at the end of an Agile iteration, the question of how to manage it largely becomes one of what scope (features) to trim from the project in a subsequent iteration. In applying EVM to Agile, how SV is managed can also drive iteration cost variance (CV) and project EAC.
In their paper presented to the 2006 Agile Development Conference, Anthony Cabri and Mike Griffiths offer an approach to managing SV. They also suggest how to incorporate AC and CV into the Agile metrics to complete the application of EVM to Agile software projects. Their approach is what we the authors have been following to apply EVM concepts to Agile SCRUM projects.
Cabri and Griffiths (2006) note that SV is only of value on an iteration basis, since the iteration represents the level at which project estimates are created. Typically, SV will not be known accurately until the end of any iteration. Also, SV is measured in feature or story point units. However, it’s fairly simple to convert SV to monetary units by multiplying SV, as a percentage of total features or story points, multiplied by the project’s BAC.
The effects of either unfavorable SV or scope changes during the course of the project can be shown by reflecting them in a burn-down chart in order to gauge their impact on the project’s end state. Converting SV to monetary units plus knowledge of the project’s resources and burn rate can easily provide a latest revised estimate (LRE) of project EAC, for example.
Using a burn-down chart like in Figure 2, and applying the SV shown in Figure 1 will track cumulative SV and show impacts of SV on the project’s end state. In their paper, Cabri and Griffiths show how SV can be used to estimate the increased project duration required to develop all features now included in the Agile project requirements backlog:
Figure 2. Burn-down Chart Reflecting Accrued Scope Increases or Negative SV
Source: Cabri and Griffths (2006)
Finally, adding a “budget burned” line to burn charts, adds the missing EVM metric AC, and additional realism to Agile measurements. It makes little sense to require the development team to maintain the AC metric in their burn-up charts, so project management can take on this task, adding AC to the burn-up chart and thus providing a complete EVM analog for Agile projects.
Again from their work, Cabri and Griffiths demonstrate how the addition of burned budget (AC) to the Agile burn-up chart, along with the addition of monetary values to the ordinate of the graph, creates a new look for the burn-up chart, which is very similar to the traditional EVM 3-line graph of PV, EV, and AC, shown in Figure 3.
Figure 3. Burn-Up Chart Incorporating Project “Burned” Cost
Source: Cabri and Griffths (2006)
By interpreting modified Agile performance charts similar to Figure 3, project managers are able to evaluate the three fundamental dimensions of EV as they pertain to the development and completion of Agile software features. Cost performance index (CPI) and schedule performance index (SPI) can also be estimated from the modified burn charts, as we will show in Figure 4. CPI can be estimated as (Completed Features) / (Actual Cost) and SPI can be estimated as (Completed Features) / (Planned Features).
Sulaiman, Barton, and Blackburn (2006) provide the most complete adaptation of EV techniques to Agile project management that we could find in the literature. They developed a set of Agile EVM metrics and a proposed a family of formulas analogous to the PMBOK® Guide’s (PMI, 2008) EVM formulas. They adapted EVM, using values defined in Scrum, and from that derived a release date estimate using mean velocity. Their paper proposes that to completely achieve Scrum’s goal of maximizing project ROI is to finish what Scrum (Schwaber & Beedle, 2001) left unfinished—fully integrate cost measurement into Scrum based on a set of logical measurements and formulas.
Sulaiman et al. (2006) applied their AgileEVM techniques and formulas proposed on two projects to test the hypothetical approach and determined the validity of AgileEVM. Their work while very interesting, is beyond the scope of this paper, which proposes that a simplified EVM approach can easily provide the project control, monitoring, and forecasting metrics missing from Agile.
Scenario and Assumptions
The authors’ experience is that Agile project teams view their work as contracting with stakeholders to deliver a series of fixed-duration “mini-projects,” each with fixed scope, that may or may not be completed at the end of the iteration. Incomplete scope is re-prioritized and re-planned into a later iteration. The question is: how should end-of-iteration variances be measured and applied to provide project stakeholders with a reasonable basis for making decisions about the final project cost, scope completed, and finish date?
For example, unfavorable SV from early iterations, unrecovered during later iterations, will drive stakeholder decisions about cutting total project scope to finish on the planned date versus incurring extra cost to finish all planned scope at a later date.
In an actual software project that was planned and delivered using Scrum, stakeholders requested EVM project performance metrics, iteration-by-iteration, including forecasts of project completion metrics such as latest revised finish date, scope completed for PV, ETC and EAC, based on project-to-date metrics at the end of each iteration. The information required was well beyond what can be obtained or inferred by looking at a burn chart.
The scenario and assumptions applicable to the development of our approach for using simplified EVM with Agile are as follows:
1. The project work was to be completed in an iterative approach and the tasks and associated effort could not be assumed to follow a linear or sequential pattern, whereby simple trending or extrapolation models for forecasting performance would not apply.
2. Project’s scope was defined at a high level and the product backlog (“feature sets”) was made available to the team prior to the first iteration (Sprint); however, the stakeholder was to further refine the requirements as the project kicked off, and re-prioritization of the product backlog was to be expected throughout the development life cycle.
3. The communication of performance metrics to non-agile stakeholders and their organization was a requirement.
4. The stakeholder and the contractor were to share project overruns and underruns, measured from the project EAC initially estimated and agreed to, at project start.
5. The project development team headcount was fixed at 10 FTE Agile Scrum-familiar developers.
6. The development effort, containing 33 stakeholder-defined feature requirements, was planned for development over five 4-week sprints (iterations).
Approach
After reviewing the literature, referenced in the previous sections, we decided to combine many of the authors’ suggestions and approaches for integrating EVM with Agile. The development team agreed to provide the burn-up chart, shown in Figure 1 throughout the development iterations. With that as the basis, and with access to weekly project cost information, we were able to produce a set of charts and metrics, using a simplified EVM approach that met the stakeholder’s requirements.
In addition to the 33-feature Agile Scrum software development effort, there were other associated project activities that were planned into work packages whose performance were being measured and reported using EVM. So we needed a reliable process to collect EVM-like metrics from the Agile Scrum development effort that would be combined with the EVM metrics from the rest of the project activities.
For external reporting and to capture EVM metrics for the software development portion of the project, we developed a simple scorecard, shown in Figure 4, which enabled us to collect and track the Scrum performance metrics and compute analogous EVM metrics. The four rows with cross-hatching in Figure 4 represent required data inputs.
Figure 4. Agile EVM Iteration Scorecard
The scorecard was updated weekly by the project manager. Figure 4 shows the status and metrics as of week #16, the completion of Iteration #4. Each week the scorecard was published in the three graphical charts (Figures 1, 5, and 6) and the EVM metrics on the scorecard were combined with the project’s other EVM metrics, to provide a complete project performance picture.
A brief explanation of the scorecard calculations may help readers understand how the scorecard was used to produce the graphics containing the following metrics:
The scorecard metrics were then reported using graphical charts shown in Figures 5 and 6. Each of these charts shows only the end-of-iteration data values, for ease of understanding. These charts were updated on a weekly basis from the scorecard data.
Figure 5. PV versus EV and SV
Figure 6. EAC versus BAC and Estimated Delay
Results
The Agile EVM scorecard and the graphical charts made it relatively easy to collect simplified EVM metrics from an Agile Scrum software development project. The metrics were reliable and accurate enough that we were confident that when combined with the EVM metrics from the rest of the project’s work packages, we could present an accurate, total project performance picture to stakeholders who were not very familiar with Agile software development methodologies.
Conclusions
Agile software development methods have proven their value during the last decade of enterprise software development by delivering cost-effective results for stakeholders faced with increasingly competitive pressures in their industries and market spaces. EVM techniques have been proven effective as a set of forward-looking metrics for monitoring and controlling complex and risky projects. While there are challenges with adopting Agile project management techniques such as feature backlogs, burn charts, and iterations to the traditional EVM view of project baselines, EV, and PV, the authors of this paper have shown that analogs for each of the EVM metrics exist and can be used to easily construct a simplified earned value view of Agile software projects.
