Organizations across industry today are on a lookout for a faster-better-cheaper way of project delivery. Thus, agile development practices have become a widespread phenomenon across all domains. Needless to say, effective agile project management for successful execution of sprints is of paramount importance. However, the root cause of most agile project failures is improper estimation, as it is directly linked to project planning, tracking, and monitoring.
The lack of a standardized estimation process definition and guidelines is one of the major challenges. The prevalent estimation unit of story point is used in a highly narrow project-specific perspective, which varies with every project and even sprints within a project. This leads to:
- Lack of predictability in estimates, resulting in incomplete sprints with hangovers in many cases.
- Faulty monitoring and control as “apples to apples” comparison of story points and velocity across sprints is not possible.
- Poor metrics calculations, leading to inability to baseline and benchmark performance and identify the right improvement levers.
Hence, the need of the hour is to have a standardized estimation model that is quick and easy to use. At the same time, this should aid in planning, tracking, and benchmarking performance without being a hindrance to the agile way of executing projects. This presentation describes Tata Consultancy Services’ (TCS) patented Agile SPACE estimation model, which addresses the above problems.
This model estimates the size and effort of agile sprints for software development/enhancement projects. Size estimation is based on pre-identified parameters that characterize the complexity of the user stories. Estimated size is bucketed into base and adjusted size. Summation of both produces sprint size. Base size depends on 13 identified parameters that determine complexity of user stories. Adjusted size depends on environmental and project-specific complexities, which may impact user story implementations directly/indirectly. The sprint size is further used to perform a capacity validation by arriving at the sprint effort. This is obtained using pre-baselined productivity depending on technology. This self-sustaining model is also powered by “actual data collection” framework and feedback adapter. Project end results in terms of actual size and effort, user feedback, inputs from industry, etc. are fed to the closed feedback loop using them.
The model intrinsically brings in a gamut of project management value-adds. It drives standardization by bringing out the breakup of the sprint velocity with reasons of assigning a higher/lower story point. Hence, person dependency is removed without changing the existing agile process of estimating story points. It enables quick and unanimous consensus of all participants of “planning poker session.” It also ensures that the same definitions of story point are maintained across multiple sprint estimations, thereby allowing repeatable and predictable estimates. It also allows baselining of sprint velocity within the project, as well as the organization and benchmarking across the industry. The same can be used to derive other metrics, which is the basis of Monitoring and Controlling Process Group in A Guide to the Project Management Body of Knowledge (PMBOK® Guide) – Fifth Edition (Project Management Institute, 2013). The inbuilt feedback adaptor and “actual data collection” framework helps in the proper forecasting of trends in terms of productivity, and defects density, thereby enabling continuous improvement.
Introduction
Since its inception, the agile way of executing projects has become an industry-wide phenomenon. Yet, a solid industry standard technique is still to be established when it comes to agile estimation. According to VersionOne 8th Annual State of Agile Survey, “the greatest concern about adopting agile methodology is “Lack of upfront planning.” This survey also indicates “improved productivity” to be amongst the top three reasons that people adopt agile (VersionOne, 2013). Since estimation and productivity measurement are the key inputs to proper planning, tracking, and benchmarking, solving these issues will pave the way for increased agile adoption.
Hence, the need of the hour was to establish a process that utilizes a standard and scientific estimation approach. At the same time, we wanted it to not be an encumbrance to the agile project execution. Currently, in agile world, there are different approaches of estimation. Some estimates are absolute (effort, time) and some are relative (story points). Yet, a singular approach with the best of “both worlds” is lacking. We tried to marry the two approaches to find out the best way to encompass both the relative, as well as absolute estimation. We envisioned an approach where a “person,” rather than a tool would participate in the “planning poker session” and recommend both size, in terms of story points for a particular story, and velocity/productivity for the sprint. The experience of this “person” would be backed by real data from past-executed projects and not just guesstimates.
This white paper explains the TCS-patented Agile SPACE estimation model, which is built around the aforementioned vision. This model not only addresses the issues involving predictable estimates for proper planning, but also helps in performance tracking and benchmarking.
Challenges
Currently, agile projects face multiple problems when it comes to estimation. Some of them are:
- Limited use of standard size estimation technique: Even for agile projects that perform relative estimation, definition of size is inconsistent. As a result, the definition of story points varies from project to project and even sprint to sprint. This makes it difficult to predict an accurate and repeatable estimate as the definition of size and productivity varies.
- Limited or non-standard productivity definition and measurement: Many agile projects do not measure their productivity at all. Even for projects that measure velocity, the method that they use to calculate the same is very specific to their project and sometimes inaccurate. Thus, the project team can never truly baseline their performance and compare the same against industry.
- Lack of structured estimation experience collection from previous delivery: Most of the agile sprints are run in silos without cross-pollination of best practices. As a result, the vast estimation experiences that exist in various pockets in the agile world are lost.
- Over estimation of sprints when there is a notion of “it has not been done before:” While performing estimations, developers within the team unnecessarily inflate the estimation in case they are working on a new technology. This mainly happens when the team does not have past projects to refer to and fears that estimating less may lead to missed deadlines.
- Lack of actual data collection framework for estimation model and productivity refinement: Most projects that are executed in agile methodology lack a proper framework for capturing metrics like size, effort, velocity, etc. As a result, the repeatability and accuracy of estimates for subsequent sprints are greatly reduced and performance baselining and benchmarking remains a challenge.
Borrowing our expertise and experience in delivering a multitude of agile projects, we have developed the all-encompassing SPACE (Story Point Based Agile Comprehensive Estimation) model which is intended toward aiding in the planning “poker session” for predictable estimates and also enabling productivity management in agile with its standardized and self-evolving framework. In this paper, we will broadly discuss the SPACE model with the help of what we have tried in order to overcome most of these challenges to help get a stronger foot on estimation and performance benchmarking and management in agile delivery.
Solution Approach
Conceptualization of the SPACE Model:
The very essence of Agile SPACE Model conceptualization and development was to retain all the positives of story-point based agile estimation and add value over and above the same for two major objectives: standardize and performance benchmark. How did we manage to do it? The answer is simple. By virtue of capturing and articulating the thought process of SCRUM teams during planning poker sessions and formalizing and bucketizing it to arrive at estimated velocity in story points for a sprint.
Making of the SPACE Model:
Though the concept seems to be a simplistic one, the model development journey was not linear. A rough sea sailing, which began with identifying the agile stalwarts in our organization -- and the most difficult fragment – and helping them understand what is bleeding in the as-is agile estimation practice and its symptom: the inability to do a performance benchmark. With several rounds of discussions and thought-sharing, our model acquired a skeleton and we started adding muscle power to it through patient capture and collate of factors. Lastly, giving brain to the model was the crucial step, which makes it intelligent enough to take inputs and provide an estimated velocity as output along with effort. Toward this closing step, we widened our collaboration circle, further involving several agile teams across domains. In great cohesive fashion, the model had been piloted, thereby imparting finesse to the same through contraction of variance between estimated and actual velocity of closed sprints. And…we offer you a standardized Story Point Based Agile Comprehensive Estimation (SPACE) Model.
SPACE Model Overview
The model is applicable for development and enhancement projects following agile methodology. The model estimates velocity of each sprint in terms of standard story points. It aids the planning poker (PP) session at the start of each sprint by providing a way to document the complexities of the user stories under scope. The participants of the PP session can select from the various complexity parameters associated with the story and the model suggests a story point for each of them. The suggestion is based on data collected from past projects and cumulative experience of agile practitioners.
This model can be used only when detailed knowledge of identified user stories are available. Typically, it is used in a planning poker session at the start of each sprint to help in its sprint planning process.
The Elaborated View
The model is aimed at estimating the following, in given order, for each sprint of a given scope of work:
- Size Estimation (in standard Story Points)
- Effort Estimation (Productivity-based)
Size Estimation:
Size estimation is performed in two distinct sections–at story level and at sprint level.
At the story level, each user story of a sprint, in scope, is sized based on three categories of parameters:
- Functional complexity parameters
- Non-functional complexity parameters
- Story-level size adjustment
Functional and non-functional parameters denote story point contribution due to the core functionalities and activities involved in the sprint; whereas size adjustment parameters denote story point contribution due to peripheral/environmental factors and/or project specific characteristics, common and static factors.
Based on the inputs provided by the estimators for the applicable parameters in three categories, the model suggests the story points in three forms -- Stringent, Likely, and Comfortable -- for the estimator to make a choice. Summation of the estimated story points of all user stories of a sprint give the BASE SIZE at sprint level.
The second section of size estimation considers adjustments at the overall sprint level. In addition to the complexity of work to be done for each user story, there can be other factors which have an increasing/decreasing impact on the overall estimated story point. These factors are taken into consideration here. A few of the many identified factors are as follows:
- Sprint duration
- Absence of SME
- External dependencies
- Large data handling
- Less business experience
For each factor, there are multiple options, which guide the estimator to identify the impact of the same and, accordingly, story point is assigned. Story point assignment is done based on historical data collected from past projects and the cumulative experience of agile practitioners.
ADJUSTED SIZE estimation is the summation of the story points of all the applicable factors having an impact on the release/sprint in question.
Final story point size estimation of a sprint is the total of the base size and adjusted size. This denotes the estimated sprint velocity.
Effort Estimation:
Effort estimation is based on productivity of the agile team and the estimated size of the sprint (as explained above). Organization baseline productivity for a set of technologies have been derived and built into the model, which can be considered to arrive at effort required to deliver a sprint, post-size estimation.
The primary productivity determinators comprise technology involved and the chosen sprints’ sequence. Baselining of project-specific productivity is recommended for each technology involved, taking into account a handful of delivered sprints spaced out evenly to cover the initial, middle, and mature sprints and actual velocity(as per model for the same set of sprints).
The baselined productivity can, henceforth, be used to arrive at effort for sprints going forward. This enables narrowing down of variance between estimated and actual effort for a sprint, thereby ensuring predictable estimates.
Model Peripherals
In addition to story point-based size estimation followed by effort estimation based on productivity, SPACE model is equipped to provide the following features:
- Size variance between estimated velocity and actual velocity sprint-on-sprint,
- Effort variance between estimated effort and actual effort sprint-on-sprint,
- Capacity validation sprint-on-sprint, and
- Sprint-on-sprint summary report to aid the retrospection sessions.
Last, but not the least, the model is accessorized with a closed feedback loop which captures feedback from users, lessons learned, and best practices and integrates the same into the framework to improve effectiveness of the same, both as an estimation and a productivity measure-and-track tool.
Exhibit 1 gives a better understanding of the model wherein a detailed process flow is given, along with the inputs to be provided and the outputs received from the model.
Case Study
Problem Statement
A technology business organization headquartered in Europe moved to an agile mode of IT projects delivery by 2012. Unlike its previous waterfall organization, however, the new mode of delivery did not come with any established metrics in place to measure the agile teams’ performance and speed of delivery. Though the organization was reaping the benefits of agile delivery like faster time-to-market and prioritized features delivery, they were not sure how good or bad they were operating when it came to saving costs through enhanced performance. Due to this lack of control over agile speed of delivery, the organization was losing an estimated US$3 million every year by overspending—15% of their yearly IT budget.
Challenges
When TCS consultants took up the matter and studied the root causes of the problem, the following gaps were unearthed:
- Agile pieces of work were not sized in a standard way across portfolios.
- Agile delivery experiences were not harvested and reused in subsequent agile programs.
- There was no mechanism to compare the speed of delivery between agile projects having similar team composition, complexity, and technology.
- Program managers were used to apply their experiences in the previous waterfall regime when it came to planning capacity. The same was absent in the new agile era.
- Going by the theory, size estimation of a sprint was left to individual team members, which was, at times, person-dependent, manipulative, and unreal.
- Actual sprint efforts were not always tracked, assuming fixed team size and sprint duration will result in fixed effort expenditure every sprint. However, actual effort expenses may not be always 100% of available capacity, which was not captured.
- In absence of standard sprint sizing technique, there was no mechanism to measure productivity.
- In absence of a defined productivity measurement approach, there was no understanding of productivity improvement processes.
The above challenges resulted in the following organization impacts:
- Significant effort was spent for every new agile program since learnings from earlier agile deliveries were not captured for reuse [resulting in initial productivity hit].
- The customer organization had less control in determining capacity for given workload.
- Absence of productivity improvement resulted in overspending on the customer's end.
Solution Implementation
The entire TCS solution implementation followed an E-D-C-B-A approach, which is to Engage with customer, Design Solution, Collect Data, Baseline Productivity, and finally, Address Gaps.
Engage
In the first phase, TCS got down to work very closely with the customer IT project portfolios, engaged with key stakeholders of the exercise, understood the ways of working, and learned the guiding principles of the customer's agile projects. The broad activities involved were as follows:
- Holding kick-off meetings with program sponsors and portfolio leads;
- Identifying a good mix of agile projects from each of the five portfolios to study;
- Studying as-is sizing process, agile principles in practice, templates, and level of documentation;
- Meeting project leads to convey the exercise vision; and
- Understanding the nature of development activities and key parameters determining story complexity.
At the end of this phase, TCS had a good understanding of the customer practices, knew the ground level reality, and could see a blueprint of the solution design, which was the next set of activities.
Design
In this phase, the TCS SPACE (Story Point Based Agile Comprehensive Estimation) Model was taken up and a fitment analysis was done with the identified projects. Slight customizations were required in the model in terms of rephrasing a few complexity parameters so that it was related well by the target audience. The series of activities that followed were:
- Based on the previous Engage phase, a detailed, eight-week plan with several milestones was drafted, as shown in Exhibit 2. The milestones were as follows:
- Milestone 1: Enlist agile projects demographics for all portfolios.
- Milestone 2: Identify ten programs across portfolios to baseline productivity.
- Milestone 3: Plan for sprint-wise data collection.
- Milestones 4-13: Baseline ten identified programs (five sprints from each program).
- Milestone 14: Formal closure.
- For each of the milestones, a detailed inputs required vs. expected deliverables were charted out
- Examples of inputs were Selection Criteria or milestone 2, Planning guidelines for Milestone 3, and so on.
- Examples of deliverables were list of total agile demographics across portfolios in designated template for milestone 1, baselining report for milestone 4-13, etc.
- Dates were assigned to completion of each milestone and the same was agreed upon within the implementation team and key customer stakeholders.
- The following team practices were setup to ensure maximum focus and high performance execution:
- Daily huddle to discuss progress;
- Continuous improvements as a daily agenda;
- Exhaustive templatization to drive standardization, reuse; and
- Fortnightly progress reports.
- Planned for a Sustenance SPOC for each program from the very beginning who will be responsible for monitoring and improving the baselines in future.
Collect
Instead of circulating a data collection template and demanding data from programs, the data collection process was positioned in a different manner. The entire focus was on accurate and limited data, rather than on inaccurate, but huge data points.
To accomplish the same, the highlight was on bringing the customer teams on the same page in terms of understanding the customized SPACE model, how it fits their requirements, and what exactly is expected out of them.
The expected information from the 10 identified programs were:
- Last five delivered sprint sizes in terms of SPACE story points (to ensure standard sizing practice).
- Sequence number of sprints for which sizing is done (with respect to iterations number in that release).
- Actual effort spent in those sprints.
- Technology combinations.
- Team headcount and sprint duration in work days.
Out of the above points, the most error-prone activity was the first one: sizing the user stories of a sprint. In order to get it first time right, the following sequence of events took place:
- SPACE model demo to identified audience and hands-on sessions with sample user stories
- Interim reviews after the first sprint is sized
- Detailed review at the end of five sprints’ sizing (where very few improvement areas were identified, owing to already conducted interim reviews, but helped boost confidence on accuracy)
Once sprint sizes were obtained, the rest of the four inputs were collected through a questionnaire (mostly available in the system).
Baseline
After ensuring that the accuracy level of collected data was pretty good, an automated template was devised which would consume incoming data and statistically generate baselined productivities for the programs.
Given similar experiences of the TCS team with other customers, this was one of the quickest phases and the baselined productivities emerged readily. In addition to that, the capacity utilization percentages were derived to comment on the improvement levers in the later phase.
Address
After a successful productivity baselining phase, the team focused on collating the results and observations to be presented to program sponsor and other stakeholders. The broad insights were:
- Program-wise baselined productivities,
- Comparative reports between portfolios,
- Suggested improvement levers for programs which are lagging behind,
- Suggested advanced levers for programs which are productive,
- Feasible productivity improvement targets year over year (Y-o-Y),
- Existing capacity utilization levels and improvement suggestions,
- Observation report on what is working well and which areas need focus, and
- Detailed productivity monitoring and improvement plan.
TCS came out at this point and the customer team was supposed to carry out the way-forward plan and recommendations for the next two years. At the end of first year, the TCS team found the following root problems were addressed after the practices were sustained. These are also the final benefits of the exercise reaped by the customer organization:
- Agile programs had a standard way of estimating size, which is person-independent and is comparable across programs and portfolios.
- Technology-wise, baselined productivities were used to estimate agile effort, which paved the way for planning accurate sprint capacity.
- Half-yearly productivity re-baselining cycles were established.
- Achievable productivity targets were communicated.
- The customer organization reaped huge price benefits of the productivity gains Y-o-Y (elaborated in the Results section), and they felt they were in greater control of agile operations.
Results
TCS provided the required support and guidance for 3 subsequent years and following is a snapshot of the returns the customer organization received as shown in Exhibits 3, 4, and 5:
Lessons Learned
As part of the post-harvesting activity, the following points were recorded as some fine prints of improvements for similar, future engagements:
✓ Few agile programs that are running for a long time without much change in team composition and business objectives tend to develop a sizing guideline, which is not completely throw-away. The same can be re-used for mapping with SPACE.
✓ The model needs to be flexible enough to accommodate sprints where a good mix of SDLC activities is not present (e.g., sprints with focus on testing only).
✓ There is a scope of introducing productivity at a story level (when size follows a non-linear Fibonacci series) in lieu of sprint level, as more complex stories tend to have lesser productivity.
Conclusion
This paper has explained, in detail, about the current challenges that exist in agile projects estimation and how the TCS-patented SPACE estimation model addresses those issues. The paper explains the model's solution approach and illustrates its usage by demonstrating a case study.
Agile estimation to date remains a grey area. Agile projects have different methods to estimate. Some projects use guesstimates, while others estimate using non-standard methods. Some agile practitioners have even gone to the extent of saying that agile sprints should not be estimated at all. Also, lack of a proper estimation approach means that an accurate performance measurement framework is also unavailable. The SPACE model addresses the dual issue of estimation, as well as productivity measurement. On one hand, it provides a solid basis for estimation using the industry concept of standardized size and productivity-based effort. On the other hand, it does so by seamlessly integrating this with the agile concepts of story points, velocity, etc. In fact, the model can be viewed as just another expert participating in the planning poker session that shares his opinion based on actual data. This guarantees a quick estimation of turnaround time, accurate prediction, and also standardization, without hindering the agile way of executing projects.