Bridging corporate and agile worlds in a large-scale SCRUM IT project
Sr. Project Manager, NTT DATA Germany
Patrick Heyder, IPMA LevelC,
Managing Consultant, NTT DATA Germany
Agile methodologies, such as Scrum are widely used and proven to be successful in IT development. However, Scrum practice is often applied to smaller-scale projects with a limited number of teams which also reside in their own setting. More and more interest applying Scrum in larger corporations and for larger IT projects is recognized. This is due to the fact that Scrum enables teams in the iterative and incremental delivery of a product, which enables rapid changing requirements.
However, large Scrum projects in a non-agile corporate environment are challenging. Different worlds collide and limit desired agility. Organizational adjustment, tailored KPIs and pragmatic methods are necessary to overcoming this conflict.
In this paper you will read about a three-year IT project involving 400 people, 85 sprints and 5,000 user stories. The authors will propose methods to overcome the gap between Scrum methodology, with its associated metrics, and the legacy corporate environment and control factors required for reporting. Common pitfalls and solutions are illustrated.
Measuring and understanding business performance of a large-scale IT implementation is crucial to success. As with any project following waterfall principles, Scrum projects in a corporate environment are similarly constraint by budget and time. Furthermore other corporate environmental factors will limit the desired flexibility and agility.
Pre-defined release windows, acceptance testing periods by external market and legacy financial and project reporting key performance indicators (KPIs) are all examples of sub-optimal conditions to establish an agile development environment. And this is particularly true for large-scale, complex implementations. Managing projects in such a context is challenging as it has to deal with an area of friction between corporate and agile project control factors.
In order to overcome this friction, it is required to bridge the gaps and overcome inconsistencies by establishing an appropriate organization to cope with complexity. This can be accomplished by introducing feature or component teams. In addition, tailored progress and financial KPIs, which are understood by different layers and levels of management, are essential. To enable this, KPIs need to be normalized across different reporting layers. Converting agile metrics such as Story Points (SP) into economically understood data is of key importance to provide transparency.
The Case-Study, a Real-Life IT Project
A large-scale and real-life project serves as a case-study to illustrate the complexity and difficulties of Scrum in a corporate environment. The project was carried out from 2010 to 2013. The project implemented a global web presence for marketing and sales for a premier car manufacturer in Germany. This included the integration of different Customer Relationship Management CRM, and other heterogeneous backend systems. It was started in Oct 2010, involving 400 project members and 15 Scrum teams, which were clustered into five feature teams. By now, 86 sprints have been completed and more than 5000 user stories rolled-out.
The project is adventurous, ambitious and it clearly revealed the challenges of applying large-scale Scrum in a legacy corporate environment.
Traditional Waterfall Approach
As with any other project, Scrum IT projects in a corporate environment are regularly validated against their performance to plan. Well known and established metrics and KPIs such as scope, schedule, cost and quality are used to accomplish status and financial reports.
In an operational traditional waterfall environment (see Exhibit 1), we can assume in many cases a legacy project reporting structure with a well-established reporting channel, structures and roles. Measurements are based on real values such as effort (hours, days, weeks), costs and resources (full-time equivalents), and its values are aggregated in the reporting hierarchy.
There is a mind shift when moving into the agile methodology for metrics and reporting. The world of metrics for Scrum teams is virtual and these measurements are relative to one Scrum team and not applicable across multiple Scrum teams. Rather than dealing with hours and Euros, Scrum teams are using virtual metrics such as story point and velocity. They use these metrics to measure and plan capacity and throughput per team. Commitments to user stories are based on these values.
While it might be acceptable to characterize one specific Scrum team through a velocity value, it is rather difficult to manage bandwidth and throughput and to aggregate an overall project status and financial report as soon as the project requires more Scrum teams to perform implementation and test (see Exhibit 2). Scaling-up with more teams complicates the reporting hierarchy and aggregation.
Virtual metrics and volatile control factors in Scrum are sub-optimal when considering commercial budget planning and schedule estimation. In a corporate environment it is generally required to be able to regularly provide up-to-date fact data regarding current status of deliverables and forecast of completion considering the overall project scope and not just the “progress to-date” information of one Scrum team. Furthermore and as projects are usually in a complex environment, they need to interact or are depending on other deliverables from other projects, which might be even using a different methodology. In this multi-project scenario it is important to establish a set of metrics supporting different methodologies by creating a hybrid KPI cockpit.
In addition to these circumstances and in particular considering a larger and larger legacy corporate environment, more challenges are encountered:
- Project reporting structures are undefined: No consistent KPIs across multiple Scrum teams.
- Project reporting channels are unclear, particularly in case of a project with multiple Scrum teams.
- Release scope is volatile: Progress status on scope completeness is difficult to assess and to aggregate.
- Legacy corporate reports on financial project status are not matching agile metrics.
- Complexity based on estimations instead of real efforts.
Requirements Hierarchy and Organization
Complex and large-scale Scrum projects require an effective organizational structure. In particular, when multiple Scrum teams are working together, clear distinction is required. In order to decompose the project scope into manageable units, which can be implemented and tested by Scrum teams, we have implemented a requirements hierarchy with themes above features, followed by user stories and tasks, which is shown in Exhibit 3.
To adequately support this scope hierarchy, so-called feature teams (FT) were established to organize work and to institute reporting structure and levels. Each feature team consisted of 1-5 Scrum teams, each consisting of up to seven team members. Exhibit 4 shows the project hierarchy and the different reporting levels. In addition to the regular roles within a Scrum team, we established the role of a so-called lead product owner (Lead PO). The lead product owner has an essential role to aggregate progress information from each Scrum team.
In this function, the lead product owner provides the bridge between corporate and agile world metrics (refer also to “Closing the Gap of Different Currencies,” later in this paper). He coordinates the different product owners in all Scrum teams of a feature team and is the link between agile and standard control parameters. The lead product owner is essentially a variance of subproject management.
Note that Scrum teams and Feature teams are inserted into the regular control hierarchy consisting of project management, middle management and top management (Exhibit 5). Each control level requires appropriate metrics to enable guidance and problem resolution in case required. The bottom of the pyramid is dominated by agile control parameters, such as Scrum team velocity, sprint burn-down and impediment lists while the further you move to the top of the pyramid, the more strategic (top management) and financially centric metrics will be reported, such as release burn-down and cost variances. The project control pyramid also shows the distinction between project related functions vs. line management functions.
Closing the Gap of Different Currencies
Exhibit 6 illustrates the same control hierarchy from a different aspect. As mentioned before a hierarchy has been inserted by organizing multiple Scrum teams into feature or component teams. Feature and individual Scrum teams have common agile reporting metrics. In other words, they use the same “currency”.
Moving on to the middle and also top management, reporting metrics are traditional and based on cost and effort. Consequently there must be a “currency exchange” established. Middle and upper level management cannot deal with virtual values, such as story points and velocity, and require hard facts in order to make reliable decisions, perform cost/benefit analysis and apply solid budget control. This area of friction is illustrated in Exhibit 6.
Exhibit 6 – Area of friction
Furthermore, and in order to provide a reliable forecasting of project completion, it is required to normalize estimates across feature teams. This does not contradict the Scrum principle of independent velocity and story point normalization per team. This is still intact and also required. Scrum teams are left independent and can establish their own reference stories and estimation rules. However, it is impossible to aggregate story points and velocities across multiple Scrum teams and to use this for schedule forecasting and evaluation of the total effort and cost required. But we all know that for budgeting reasons forecast of cost and schedule is required, even when following a Scrum approach.
To overcome this dilemma, normalization is required to enable aggregation. This normalization is happening on the feature level and is carried out by the lead product owner.
Exhibit 8, Exhibit 7 and Exhibit 9 show typical example reports used to manage a single Scrum team. They show well known velocity, sprint burn-down and impediments backlog charts. Velocity graphs help to identify the team's speed of development, enable forecasting for sprint planning and progress monitoring. Sprint burn-down charts give overview and trends for user story completion and helps to forecast achievement of the sprint goal. An impediment list and backlog provides an overview of distraction that happened during a sprint and therefore gives an indication of effort spend and required (forecast) for unplanned and non-Scrum activities.
Moving on to the feature team level, aggregated metrics are used and bridge the gap between virtual and fact data. The velocity per feature team (Exhibit 11) uses a normalized metric across multiple feature teams and by doing this enables the comparison of velocity between Scrum teams. This is of key importance.
Effort analysis and forecast (Exhibit 10) enables monitoring project progress and forecasting trends for each feature team and it allows comparison of remaining effort (in hours or man-days) compared to planned value.
Note that at this point we have left the pure world of Scrum metrics and are reporting fact data as known by most non-Scrum people and as they are required in a corporate legacy reporting and control environment. But how are we converting and calculating for example remaining effort, which is necessary to report progress and make forecast estimates? Here, the lead product owner comes into play. In order to provide effort estimates in hours/days the lead product owner aggregates the specific progress and performance data from each Scrum team within the feature group. He or she does this on an empirical basis using the user-story priorities from the backlog, the current and forecasted staffing per Scrum teams and the Scrum team velocities. By performing a correlation between these values the lead product owner performs an estimate of remaining effort and reports this based on features.
Looking at team efficiency, a report such as illustrated in Exhibit 12 is helpful. Here we can perform a comparison of cost vs. remaining effort per sprint and consider corrective actions, e.g. by removal of impediments. This type of report obviously requires, besides the estimation of remaining effort, additional information about user stories and features as well as any associated impediments. With this data, a report as shown in Exhibit 12 can illustrate the current ratio of cost and effort and help to define corrective actions.
Additional Dashboard Metrics
Note in general that each level of management needs the right tools and metrics which are understood and provide enough guidance to make solid decisions. Exhibit 13, Exhibit 14 and Exhibit 15 provide additional examples of metrics considered on middle management level.
In comparison to Exhibit 10, which illustrates effort consumption and forecast, a feature burn-down chart is shown in Exhibit 13. The step curve describes the completion rate for features for the overall project. A step in the graph depicts 100% completion of a feature. Partly completed features are not shown. This is done on purpose to make sure that only completed features are tracked. This avoids the typical “90% completion syndrome,” which causes uncompleted work to be shifted over and over in time and never getting completed.
Exhibit 14 shows a classical cost variance chart and is useful to perform cost-control in the project. This is particularly important in a corporate environment, because here projects are mainly constrained by budget limitations. Budget variations may quickly occur and therefore a comfortable reporting of forecasted cost is critical. Note that this is typically not so easy following a theoretical Scrum approach with virtual metrics only.
Looking forward and taking into account the role of the lead product owner, a so-called ‘Release Burn-up’ chart is useful (see Exhibit 15). This metric provides information about the forecasted resource capacity in order to implement the next major release in correlation to the backlog. In addition it gives indication about the utilization of the teams in the future in order to forecast the resource capacity required.
Understanding the different management levels and “currencies” used to control and guide project execution is of key importance when working with large-scale Scrum projects in a corporate environment. Legacy systems and control mechanism require legacy fact metrics. Scrum methodology uses virtual metrics, but those metrics are inadequate to be used for report into the middle and upper level management hierarchy. Consequently, a currency exchange or transformation of metrics must happen. This can be accomplished by establishing another layer of hierarchy (feature teams), establishing the role of a lead product owner and normalizing metrics on this level. This bridges the gap between Scrum and corporate legacy worlds and enables appropriate project and budget control and forecasting across the complete hierarchy. In addition, various other reports were discussed, which can be used as tools to visualize status and control a large-scale IT Scrum project residing in a legacy corporate environment.
Besides the challenges discussed in this paper, other difficulties need to be considered for large-scale Scrum IT projects. Those are similar to issues encountered in smaller Scrum projects. However, the impact is larger and therefore the issues need to be cautiously taken care of. The existence of a well-defined and agreed-upon DoR (Definition-of-Ready) and DoD (Definition-of- Done) for Scrum deliverables is key critical when multiple teams are working together and depend on each other. In addition, an organization needs to be ready for Scrum and needs to realize and understand the different worlds or control metrics, as described in this paper. Communication among Scrum teams within a feature team group and among feature teams need to be established and formalized.
Bauer, M. (2011). How to succeed with SCRUM. Retrieved from http://www.martinbauer.com/Articles/How-to-Succeed-with-SCRUM
Bauer, M. (2011). Is agile cheaper than waterfall. Retrieved from http://www.martinbauer.com/Articles/Is-Agile-Cheaper-than-Waterfall
Leffingwell, D. (2011). Agile software requirements. Upper Saddle River, NJ: Addison-Wesley
Leffingwell, D. (2007). Scaling software agility. Upper Saddle River, NJ: Addison-Wesley
Pichler, R. (2007). SCRUM - Agilesprojektmanagement erfolgreich einsetzen. Heidelberg: dpunkt.verlag
Schneider, S. (2013). Herausforderungen für automotive SCRUM. Retrieved from http://www.sebastian-schneider.eu/cms/agile/funktioniert-SCRUM-im-automotive-bereich-wirklich-nicht
This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI or any listed author.
© 2014, Thomas Zimmermann, Patrick Heyder
Originally published as a part of the 2014 PMI Global Congress Proceedings – Dubai, UAE