Metrics for agile projects
finding the right tools for the job
Agile project managers have a wide range of metrics to choose from and that show different types of progress at the iteration and release levels of agile projects. A deeper understanding of the variety of metrics for the agile methodology will enable project managers to determine how and when to use them most effectively when communicating progress to project stakeholders. This in-depth presentation will provide the best practices and examples from the presenter's real-world agile implementations to enable attendees to select the right tools for the right job.
The discussion will begin by reviewing the metrics attendees currently use to track and report progress to stakeholders in a project, iteration, and release. Details for iteration planning and estimating will build the framework for selecting progress reporting metrics. Best practices for progress reporting will include a detailed examination of how and when to use the methods available, including daily meetings, iteration delta tables, release and iteration burn-down charts, progress reports, running tested features, and earned value management. The value of tracking iteration back to release will also be explained.
Through this interactive guided discussion, project managers will learn how to expand the use of the metrics at their disposal to fine-tune the progress made on their agile projects.
One of the key strengths of traditional project management methods is that they provide a robust, tested toolkit of metrics and reporting techniques that grant stakeholders visibility into the progress and challenges of the development effort. By measuring progress against the Gantt charts developed during project planning, by ticking off tasks on the work breakdown structure as they are accomplished, and by monitoring issues logs and defects (or “bug” lists), stakeholders can keep a finger on the pulse of the project and set their expectations appropriately. Modern project managers add indicators, like project dashboards, to further simplify the tracking of projects by refining the main points of progress and challenges into simple, easily parsed graphics.
Due to the nature of agile projects, many of these familiar metrics are not available. Because agile project managers are not developing long-term, multi-month project plans with predictions that reach far into the future, the familiar Gantt chart and work breakdown structure are rare in agile programs. Agile project managers utilize other metrics, such as burn-down charts, to understand where their projects stand. The project manager is responsible for the overall delivery of the project's objectives and performs this work by keeping the team focused, encouraging teamwork and individual productivity, ensuring that the team has the resources required to be successful, and managing the customer relationship as the product is built. Agile project managers are not focused on updating Gantt charts or keeping the project dashboard current; instead, their focus is on keeping the performance of the team at the highest level of productivity and collaboration and isolating the team from distractions, barriers, and digressions that could harm their concentration. Because team members in agile environments are expected to manage their own workloads, project managers serve to ensure that their teammates have the information, resources, and technical coaching they need to be successful.
Tools for Tracking Progress
During the building of an iteration, while the developers are focused on creating the features they have committed to delivering, the project manager is responsible for understanding the progress that the team is making and keeping the stakeholders informed of the project's progress. Exhibit 1 is an example of a chart that project managers can use to communicate iteration status to the customer and stakeholder community. Designed to communicate the changes from one iteration to the next, this chart helps stakeholders see what the original plan for the current and next iterations looked like a couple of weeks ago and how they look today. Features that are new, have been deleted, and/or have been moved from one iteration to another are all clearly indicated.
Another tool that many agile project managers use to track progress is the “burn-down chart,” illustrated in Exhibit 2. The project manager starts at the upper left-hand corner of the chart, before any development work is completed and as features are completed or “burned down,” the project manager tracks that progress with a simple line chart. In this example, the project manager is tracking a release, which contains a total of 200 features, or story points, that the team must “burn through.” A burn-down chart can also track the progress of an iteration by substituting “days” for “iterations” on the y-axis. As features are built and tested, the number of story points remaining is plotted and a line connecting each point is drawn. By keeping the burn-down chart current, project managers can offer their customers a quick, visual representation of the functionality that has been delivered and is left to be delivered, and provides clues regarding the team's productivity.
If the chart is a flat line and not moving down toward the zero point in the lower right-hand corner, it shows that the team is stuck and not able to make progress as expected. It sends a signal to the project manager that a technical issue, an organizational barrier, or a teammate's productivity requires attention. If the plot moves upward, it indicates that new features have been added to the backlog, or features previously considered complete have moved back to development. In agile development, a new feature added to an iteration can mean many things: it can be a positive sign of fruitful interaction with the customer or it can be the first indication of scope creep or project gold-plating (in which team members add features because they're “cool” or “elegant”).
An upward tick in the burn-down chart signifies that a feature previously considered complete has moved back into the unfinished column and is a negative indicator. Features should only be burned down on the chart if they are developed and tested and ready to be integrated into the iteration or release. Counting unfinished features creates all sorts of complications, from damaging internal team trust to corrupting the entire burn-down process. Project managers should set clear rules regarding the state of completion required before a feature or story point is counted as burned down.
The burn-down chart is also very revealing in regard to the team's estimating success. If the plot is proceeding as expected, and the trend line is an even, steady descent from the top left-hand corner to the bottom right, this signifies that the team has been extraordinarily successful at predicting its velocity and selecting the right features and right number of features to include in the iterations. If, on the other hand, the line is choppy, sloping up and down and stalling frequently, it can indicate that the team is not cohering efficiently, or that unforeseen complexities are arising that must be addressed. It should be clear that, in all of these circumstances, the project manager and the team can use the burn-down chart as a key indicator, not just of progress, but of the internal productivity of the team.
It should be clear that, as developers go off to do the technical work of implementing the features for which they have signed up, there is the risk of creating little islands of technology, which do not integrate well into a coherent product. This risk, of course, exists in traditional and agile projects, and the collaborative techniques of agile project managers should assist in minimizing integration problems. Still, agile project managers must guard against this contingency, and the mechanism typically used for this is often referred to in the agile community as “frequent integration.” Most experienced project managers can cite an initiative in which all of the separate elements were tested and performing appropriately, only to find that, when put together, the interaction of the parts created unexpected issues, consequences, and even failures. In fact, many projects are thrown off track by late integration challenges, which can be very difficult to uncover and repair. Although integration is not a feature and will not show up in a task allocation session or on a burn-down chart, prudent project managers ensure that they have allocated time in their iteration cycle to pull the separate components of the design together, as they are created, and that they are working as a unit. Some agile teams assign a single teammate to be the integration manager, with the responsibility of reviewing components as they are completed and ensuring they fit together.
Although the burn-down chart can provide an at-a-glance overview of the team's progress, it isn't enough to help the team stay on top of the project effort. In most agile approaches, teams use a variant of the daily meeting to track the daily issues, risks, and challenges that arise. These sessions are often called “stand-up” meetings, because agile proponents believe that by standing rather than sitting around a conference table, teams are encouraged to keep these meetings short, concentrated, and effective, which helps to avoid the plague of irrelevant and endless meetings that can distract from progress. In the scrum method (named for the players' huddle in rugby), the team stands together and, by going one by one around the room, answers three simple questions:
- What have you done since the last scrum?
- What will you do until the next scrum?
- What issues, risks, or barriers have arisen that could distract you from accomplishing the iteration goals?
Other agile methods have their variants of the daily meeting but all are focused on planning and communicating at the granular, daily level. Some standard characteristics of a daily meeting are: each meeting is 15 to 20 minutes long; everyone stands in a circle; each meeting occurs at the same location; the order of the presentation is defined; team members share status/obstacles; and, all team members are invited, but only people involved in the iteration speak at the meeting. This meeting is the forum for discussing the tasks that slipped or were completed early, the unexpected barriers or challenges that arose in the previous day of development, and for refining plans on a daily basis, depending on the realities of this particular effort and team. Features and tasks can be swapped by team members as they discover their own suitability for the tasks they have accepted. Schedules can be refined as the team's velocity becomes clearer. This daily team meeting often is referred to as the heartbeat of the agile methodology.
Agile project managers have an important responsibility to the customer: they must coach the customer on the responsibilities of acting as the organization's representative on the development team. This is often a new and unfamiliar role to customers, who are accustomed to developing a set of requirements and then waiting for the product to appear from the development laboratory. The intimate, continuous participation of the customer in the development process and in the decisions made is a key precept of agile development, but this does not mean that the customer is prepared to play that role effectively. Strong agile project managers help to mentor customers through the process, bringing them into the iteration planning sessions, inviting them to the daily meetings, and helping them to understand project status by teaching them to interpret the burn-down charts that the team uses to track its progress. The role of the customer in an agile project is demanding, and the project manager must ensure that customers understand and fulfill their role. Without the constant input, reaction, and support of the customer, the agile effort will flounder. The project manager must ensure that this disconnection is avoided.
Iteration status charts, burn-down charts, and daily stand-up meetings are effective progress reporting tools, but can sometimes be obscure to the uninitiated and do not answer stakeholder questions about value delivered, costs, and resources consumed or future expectations. Although agile theory believes that customers should be so intimately involved in agile development efforts that they know as well as the actual developers where the project stands at all times, in practical terms, this is rarely the case. And, even if it was the case, customers may not have the skills or language to pass that understanding on to the entire stakeholder community. Clearly, it is critical that agile project managers devise strategies for tracking and measuring the team's performance on the project and for reporting that progress to the customer and stakeholders.
One of the key philosophies of agile development is that the elements that are not directly tied to developing the features of the product, such as documentation, reporting, administration, and measurement, are to be kept “barely sufficient,” just enough to deliver the value that the customer is expecting, but with a minimum of overhead, complexity, and the ceremony associated with them. This philosophy is applied in metrics as well. In many agile teams, the burn-down chart, or some variant, is the sole visible metric of progress. Other teams go to the opposite extreme and attempt to apply all of the artifacts of a traditional methodology—such as a work breakdown structure, Gantt chart, earned value management, and dashboard—to agile projects, even if they only apply to one iteration at a time. Although most agile adherents would resist the adoption of the entire suite of traditional project management office-style metrics, most would also agree that something more than the burn-down chart is required.
For those teams coming from a traditional project management environment, familiar techniques like earned value management, with an agile twist, can bridge the gap between project management office–style practices and more agile methods. Earned value management is a project measurement technique used to evaluate and predict project performance against the plan. Project managers training in the Project Management Institute (PMI®) process will recognize this technique, which measures three elements:
- Planned expenditures
- Actual expenditures
- Actual work performed against planned expenditures
This last element, of course, is the crux of the impact of earned value management. It measures the value of what has been delivered for the amount of budget spent. Its underlying concept is obvious: If you have spent 80% of the budget and only delivered 20% of the value, you have a productivity problem. If you have spent 20% of the budget and achieved 80% of the value, you've got plenty of slack in case of contingencies, and you also probably have a serious problem in your estimating process.
Formal earned value management is a rigorous process, measuring components such as:
- Planned value, which represents the budgeted cost of a defined amount of work; for instance, how much was budgeted for the work expected to be done in the next three weeks
- Earned value, representing the budget for the amount of work expected to be done during a defined time period
- Actual cost, the real cost of delivering the actual work accomplished in a specified period
These metrics are then used as the basis for calculations that give project managers guidance as to the performance of their projects. Calculations like cost variance, schedule variance, and estimate to completion can be derived from various formulas (PMI, 2008, pp 182, 184–185).
Although the method of calculation seems rigorous and disciplined, there are those who are skeptical regarding the application of earned value management; because it claims to measure delivered value, some doubters question whether an incomplete project delivers any value at all: of what value is half a runway or three-quarters of a software application? This explicit connection of tasks completed with value would meet with tremendous resistance among agile proponents. Notwithstanding the cynics, earned value management is a widely accepted project measurement technique. Can earned value management be used with agile techniques and, if not, how can a rigorous measurement be applied to agile project processes?
In traditional project approaches, it is assumed that the scope is fixed, any changes will be managed through rigorous change control, and the project will proceed in a predictable manner in which past performance is indicative of future progress. Earned value management calculations are made based on a detailed work breakdown structure, and project managers focus on areas out of compliance to keep the project on track. Managers and customers rely on earned value management to understand project performance and to manage teams, contractors, and vendors.
In agile projects, many of the underlying assumptions are not applicable. Project scope is assumed to be defined broadly, and only the current iteration is planned and estimated in detail. Changes are expected as the product is refined and optimized, making it difficult to baseline a project.
Applying Earned Value Management to Agile Project Management
To apply measurement techniques like earned value management to agile project management, project managers should focus on the expected outcome rather than the method. The use of burn-down charts as measurement and reporting tools provides many of the benefits of earned value management, but in a different form. If the burn-down charts are kept scrupulously current and analyzed prudently, project managers can use them as a key indicator of project progress, problems, and risks.
Some agile proponents have taken the application of earned value management (EVM) to agile programs to a higher level of rigor in a method they call, appropriately enough, AgileEVM. In a paper presented to the IEEE, a team of agile developers had proposed a disciplined, quantitative technique for applying earned value management to agile methods (Sulaiman, Barton, & Blackburn). AgileEVM uses story points, rather than tasks performed, as the basic unit of measurement and measures iterations planned against iterations completed to derive the value delivered.
As in earned value management, AgileEVM (Exhibit 3) requires initial baselines, such as number of planned iterations, number of planned story points in a release, and planned budget for the release. Also needed are the total number of story points completed, number of iterations completed, actual cost, and the number of story points added or deleted from the release plan.
For example, the budget for the release is $100,000 for a completion of 100 storyboards. At this time, you have completed 25 of the storyboards at a cost of $20,000.
Actual percent complete = 25 completed storyboards/100 storyboards = 25% complete
Earned value = actual percent complete X total budget = $25,000
These modifications make the version of earned value management more compatible with agile techniques while providing a familiar terminology and reporting mechanism.
Agile projects do not necessarily need to reinvent every artifact; traditional status reports can be used as long as it is clear that they refer to the current iteration rather than the entire effort. A typical status report, with an executive summary, accomplishments report, listing of upcoming activities, and issues/barriers list, can be quite effective and has the added advantage of being familiar to the audience. As long as the project team stays focused on completing the development work and delivering a working product or prototype rather than being distracted by status reporting, traditional methods of communicating progress can be effective.
It is important to remember that many executives, stakeholders, and customers are steeped in traditional project reporting methods and can feel unconnected or uninformed during the migration to agile techniques. Customer-focused agile project managers work closely with their clients and sponsors to develop reporting mechanisms that tell clients what they need to know and inspire confidence in the team through transparency and communication.
Project Management Institute (PMI) (2008). A guide to the project management body of knowledge (PMBOK® Guide—Fourth Edition. Newtown Square, PA: Project Management Institute
Sulaiman, T., Barton, B., & Blackburn, T. (2007, July). AgileEVM: earned value management in scrum projects. Proceedings of Agile 2006 Minneapolis, MN.
© 2010, Nancy Y. Nee
Originally published as a part of 2010 PMI Global Congress Proceedings – North America