Abstract
Agile is in fashion, and this trend has also reached large corporations. But how can agile, which was designed for small software companies, be implemented in an environment where (1) the waterfall methodology is deeply integrated; (2) where budgeting is based on the waterfall method; (3) where management has no idea what agile is; and (4) where project deliverables are regulated by authorities?
This paper covers how agile can be successfully integrated into an existing waterfall methodology, so that buy-in from stakeholders (project managers, quality personnel, finance and management personnel) can be gained. It also shows that agile is not a silver bullet and that elements from agile can be combined with waterfall to create feasible solutions.
This paper is directed to members of project management offices who can directly influence a methodology.
The focus is on the following areas:
- How to integrate agile into an existing methodology in relation to:
- phases
- milestones
- documentation
- roles
- financial processes
- How to deal with regulatory requirements in an agile project
- How to gain the management buy-in to initiate agile implementation
- How to roll out agile to get the project managers interested but also to ensure that the agile project will be successful
- How to establish important factors for successful agile projects in large corporations
The Struggle to Introduce Agile in Large Companies
Four key factors make it difficult to introduce agile in large companies:
All Procedures are Aligned with the Waterfall Methodology
In most cases, when a project management office considers introducing agile within a large company, there is already an existing waterfall methodology in place. All templates, tools and training materials are aligned with agile, and all processes serve as interfaces with other departments. Good examples are the financial and communication processes. Introducing agile cannot be accomplished as an isolated effort from the project management office. Many other functions have to be involved about the value to create new processes.
Management Needs to be Convinced
In a lot of companies today, scarce attention is given to the details of project management. In addition, managers like to be able to know exactly what to expect. So to approach management with something new, which is more flexible and talks about the self-management of teams, does not always trigger a lot of enthusiasm. It is crucial to know what to ask for and to have the right arguments ready.
External Regulations Destroy the Agile Spirit
The biggest enemies of agile are rules and guidelines, which limit flexibility within projects. Good examples are GxP in the pharmaceutical industry or SOX in the financial industry. These regulations ask for copious documentation and take a lot of the agility out of agile. Are there ways to still use agile in such projects?
Many Project Managers Have the Wrong Perception About What Agile Is
This point is surely not exclusive to large companies, but because project managers in large companies usually have not had much exposure to agile, it's a bigger issue. In short, there is the misperception that agile means less or even no documentation, and the same goes for control. So they imagine agile to be the project manager's paradise: Just start somewhere, do whatever the customer wants, without writing anything down and without taking time or money into consideration. This misperception initially causes excitement among project managers when they hear that agile will be implemented and often leads to greater frustration when they discover the truth. Expectation management is hence a key part of the implementation actions.
Integration of Agile into an Existing Methodology
Waterfall methodologies in large companies are in most cases deeply embedded in processes which link to other departments and which are outlined in various guidelines and depicted in IT tools. The attempt to create a completely new methodology with new processes which fit perfectly with the methodology will in most cases fail. As nobody is willing to do all the changes that would be needed, the consolidation of various outputs for reporting reasons would be difficult and the roles available would be different from those needed for agile. Therefore, we have to abandon the dream of using “pure” Scrum as an example. It is more critical to integrate our agile approach into the existing methodology, so that it has only minimal impact on all existing processes.
Phases
For the following section I will use the five project phases that A Guide to the Project Management Body of Knowledge (PMBOK® Guide) suggests:
- Initiating Process Group
- Planning Process Group
- Executing Process Group
- Monitoring & Controlling Process Group
- Closing Process Group
So how can we modify phases so that they can also be used for agile? The solution is to leave the phases which frame the project untouched as much as possible. These are the Initiating, Planning and Closing phases:
Initiating
For the Initiating phase this is rather easy: Doing a business case and a project charter is recommended in most agile methodologies, so no modification is needed. Also rough estimations on time and resources can be made, and the basic scope of the project is also known. The project can begin almost exactly as if it would be a waterfall project.
Planning
The Planning phase is the phase where waterfall and agile differentiate the most and it seems hard to bring both worlds together: product backlog vs. requirement specification–what to choose? The solution is to use both. In a first step the requirements are gathered as usual, with the difference that it will only indicate the initial intent of the customer, and not serve as the final requirements, which cannot be changed anymore. In a second step these requirements are translated into user stories and filled into the product backlog. Changes which occur during the project will only be recorded in the product backlog, with the exception if the original scope is substantially changed. At such a point, the requirements would be revised and signed off again.
Closing
The Closing phase can be handled in the same way as in a waterfall project. This starts with the final release and continues with all that is included in this phase like the final testing, handover to operations, lesson learned and obviously also a celebration!
The iterative phase: Executing and Monitoring & Controlling
Having covered the “shell,” what stays is the iterative core, which cycles during the sprints within the Executing phase and proceeds for each release through the Monitoring & Controlling phase. This is the area which is usually the least affected by processes influencing other departments and we can therefore implement here all the agile ceremonies and artefacts without significantly disturbing the existing environment.
Milestones
As milestones are used to receive a holistic overview of a portfolio and are also perceived as a means to ensure the quality and health of a project, it is quite crucial that even in an agile project they don't differ from the way they are used in a waterfall project. And once again this is not that hard to accomplish. Most milestones are usually lying somewhere within or at the border of the “shell” phases. So we can leave them untouched. Within the iterative phases, existing milestones will simply be repeated once for every release.
Documentation
Documentation can be divided into three categories–untouched, open, new:
Untouched
This is the easiest category–these are documents we simply use in exactly the same way as in a waterfall project. Good examples are the project charter or a project closure document.
Open
A lot of documents, which are used in a waterfall project in a way that they are written once and then signed-off on, can also be used in an agile project, just without the sign-off. These documents are written in an initial version at the beginning of the project, and then revised after each release. Examples are the test plan, release plan and operational handover documentation.
New
The third category is the agile documents and tools–also called artefacts–which are added to the existing documentation. It might be advisable to adjust them from a look and feel point to match the existing documents. What is nicely visible here is that the new agile route will require more documentation than the waterfall route, because we add documents to the existing set, and there is not much we remove. It is important that we keep this in mind and do a clear expectation management and are also prepared to justify why this makes sense at trainings.
Roles
Roles are usually aligned with the waterfall methodology; job descriptions are based on these roles; and people are hired to fill these roles. To whom can we now assign roles like ScrumMaster, which did not exist before? Important is not to prescribe a solution but much more to give options. For example, to stay with the Scrum method, the product owner role can be taken over in a small project either by the project manager, or by a business analyst or an operational application manager. Depending on the roles available, there may be multiple options for each role.
There is only one obvious “Do” and one “Don't” in my opinion:
Do: Always have a project manager assigned for each agile project. Don't make the mistake to believe that just because in Scrum this role is not available, you could do without it. In large corporations there are a lot of tasks in the areas of finance, communication and reporting, which are not foreseen in Scrum, but still have to be taken care of.
Don't: In general, it is not a good idea to assign a project manager the role of a ScrumMaster. These two roles do not match at all and the worst combination is if a person should be doing both roles at the same time. This is a clear recipe for failure.
Financial Processes
There is almost nothing more regulated and inflexible in large companies as financial processes. To get a project approved without clearly stating what the money is needed for is mission impossible. If we want to follow agile, we should be able to state: “We intend to use for the following solution, which we describe in high level, an amount of xyz, and with this money, we will implement the features that have the highest priority for us.” There are three options to achieve this:
New process
If there is one area where to introduce a brand new process is worth fighting for, then it is finance. Because only a clearly agile designed financial process can give the project team freedom to embrace the methodology and execute it in an ideal way without being slowed down by unnecessary administrative roadblocks. It might be difficult to convince management and the financial department that this is necessary, but it will pay off. Key is that the customer can allocate a sum to a project without linking the amount to the exact requirements, as they are simply not known yet in an agile project at this stage.
Based on initial requirements
If a new process is out of the question, the original budget could be requested based on the initial requirement specification, while clearly knowing that it will change. This solution might do the trick but is definitely in the grey zone. The question here is how strict the rules are–would it be allowed to use the money assigned for one feature which will not be done for another one?
Release budget
The least favourable approach would be to create for each release a separate budget right after release planning, which then would be approved before the next release cycle starts. This approach, while it is the easiest to implement, has two big issues. On one side, it might create a bottleneck as a release cannot be started before the budget is approved. On the other side, as the number of releases or the total budget is not specified, it might lead to an open-ended project.
Regulatory Requirements
One of the big advantages of agile is that with it projects can deliver working software frequently and hence the stakeholder can profit from newly developed features as soon as possible. So far so good–but what if each rollout takes an extreme amount of time and resources? Exactly this can happen when projects are conducted within a regulated environment like the financial or pharmaceutical industry. Advice which is often heard is to simply not use agile in such a situation. Nevertheless, two options exist in which using agile would be feasible.
Validation of Each Release
If each release should be rolled out into a productive environment, then this is the only way to go–each release has to be validated, which means applying all the measurements that the regulations require. There are two things that should be considered. The first is to understand thoroughly what the absolute minimum requirements are and to only do this during the releases. Anything that makes sense, but is not absolutely required can be done at the end of the project. The second thing is to find the right amount of releases. Validation of each release requires a lot of effort. The fewer releases planned, the longer the release cycles will be.
Retrospective Validation
The second way to proceed is to roll out the various releases without validating them as usual, and then at the end validate the whole solution retrospectively. This saves a lot of time, but can only be applied if the releases are not used in a live environment. We lose with this method the advantage to deliver fast working software, but we keep the high customer involvement which ensures that the final solution contains the features that have the highest priority to the customer. And we also can roll out the final solution faster as when we validate each release separately.
Obtaining Management Buy-in
The key decision that will lead to a failed agile introduction is mostly made right at the very start. It's when the project management office goes to senior management and proposes to do an agile “pilot” to prove to them how great agile is. These pilots are prone to fail because project managers will not go through the effort of implementing agile into their own methodology and thoroughly training all the involved project managers and other stakeholders.
Instead, they take Scrum and decide that a project manager who has done a Scrum training once will do a project in agile. This project will fail, because Scrum will collide with all the existing processes. Nobody understands what is going on and it will prove once and for all that agile brings nothing but chaos. And with that, the topic is off the table.
So the project management office (PMO) has to be able to convince the management that agile should be introduced as an additional option–and only when the management is convinced should this endeavour be started. If the management wants prove that agile brings value, a much better option than a pilot is to get a success story from a similar-sized company in the same industry.
Rolling Out Agile Successfully
As with rolling out any methodology, one of the key actions to be successful is to ensure the users understand how to use the methodology. There can be various measures taken to ensure this:
Training
To train users adequately is the most important success factor. A full day of classroom training should be the minimum. I propose using three elements in the training: First, explain the “pure” method to the participants on which the adaption is based (e.g. Scrum)–they don't have to be experts afterwards, but they should have an idea of what this method is all about. In a second step, explain how the method was adjusted to fit the corporate methodology and how the users should apply it. As a third part, it is recommended to include games, which demonstrate in a hands-on way the advantages and the processes of agile.
Mentoring
In the same way driving a car cannot be learned only by theory, it is also an illusion to believe that after a classroom training–no matter how long–a project manager can conduct his/her own an agile project without any support. Many questions will only appear once the project has started. Offering some support by mentors at this stage is vital. This can be achieved at the start with external support and later on by experienced internal project managers.
Communicating Key Points
Besides explaining agile to project managers, there are a few key points which have to be communicated:
- Agile does not mean less documentation, but can even mean more documentation.
- Agile does not mean conducting a project in an uncontrolled way.
- Agile is no silver bullet, and every project must be carefully evaluated to determine whether agile should be applied.
- Agile is substantially different from waterfall–project managers who want to apply it have to invest time to learn and gain experience with it.
Important Factors for Successful Agile Projects in Large Corporations
Mutual Trust
Politics is rather widespread in large companies, and departments or regions sometimes compete or even fight with each other. Wherever there is no trust, there is also no environment for agile. Agile requires an environment where the project team can trust that the customer will communicate what they need openly and what constraints are existing. And the customer needs to trust that the goal of the project team is to deliver the best possible solution for the customer.
Readiness of Customer to Invest Time and be Involved
Agile requires the customer to be involved almost daily in the project. This should not be taken as a given, as first of all, customers will not be used to such an intense involvement, and some of them also do not consider this as positive. There will always be a share of customers who will prefer the “I tell you what I want and once you're finished you might show it to me” approach. So before choosing the agile route, the willingness of the customer to invest the required time should always be checked.
Team Spirit
While in a waterfall project hierarchical management style is still predominant, in agile teams are self-organizing and everything depends on team members who are motivated and want to work with each other. This means that a team has to be selected much more carefully. A case in which this is not possible and team members are simply assigned to a project by management, a careful evaluation if agile could work with this particular team is advised.
Collocation
Collocation is quite an important requirement when it comes to agile. But this does not mean that an agile project can only be conducted if the whole team is collocated. What is crucial is that the ScrumMaster and the development team are collocated. All other involved team members and stakeholders can be involved with other means, like video or telephone conferencing, or by sharing computer screens.
Conclusion
Implementing agile within a large company is feasible and will bring substantial added value to the customer and the company as a whole. But the task to implement agile should not be taken lightly. To gain management buy-in and to adjust agile to existing methodology and to existing processes, is key to be successful, even when this means to leave the path of pure agile. And still, the main focus should always be on the project teams. Only if they see a benefit in using agile and only if they understand clearly how to use it and have a guiding hand in their first attempts, this new route in managing projects will be embraced and used by them.
