Project management and agile

Primavera Systems, Inc. is a leading project and portfolio management software company. Founded in 1983, Primavera is a diversified software company that provides both desktop and enterprisewide solutions that enable companies to prioritize, plan, manage and measure the business and services of their customers. With a Philadelphia-based development group with more than 100 programmers, testers, documenters, business analysts and project managers, the company's solutions have been used to manage projects worldwide that total more than $5 trillion.

Primavara's unwavering mission is to eliminate project failure. Our goal is to create a world-class organization and be recognized for developing the highest-quality software and services – as well as the way we develop them -- to meet the needs of our customers around the world. We also want to create an environment of excitement, continuous learning and success for our employees.

We know that to continue to meet our goals, we must be able to adapt and change with today's dynamic marketplace. Our most recent undertaking was the addition of Agile Software Development to our traditional project management environment, which has given new enthusiasm to our development teams, and brought a new momentum to the way we develop and deliver ever better products to our customers.

A Look Back: The Waterfall

For most of our first 20 or so years, we developed software according to the traditional waterfall model whereby projects were requested by our Product Marketing Department, planned out and developed over a period of nine to12 months, and delivered to customers at the promised time. However, meeting that deadline usually took its toll. The closer we got to the deadline, the more last minute changes came in. Defects multiplied, and programmers were relegated to working more and more overtime – a frenzy that continued to pick up speed until we released the software, and we all celebrated our accomplishment.

But, the celebrations were short lived. Our rush to completion meant that the new software didn't have all the features that were requested. Further, because Product Management hadn't seen a finished product until the end, and with no process in place for the stakeholders to be involved in the development process, their expectations had become disconnected from the mission of the development team.

Our business was growing rapidly and we knew that the needs of our customers would continue to expand as they responded to their own changing environments. Our challenge, then, was to find a new way to work that would allow our product development efforts to evolve along with our customers'. We would have to become more flexible, more nimble, more responsive to change – in short, more agile.

Discovering Agile Software Development

Agile Software Development's roots are in lean manufacturing, which was introduced by Toyota in the 1950s. Lean manufacturing mandated a radical change in the company's business model. Instead of pushing more products into the marketplace, why not let customer orders drive production? While lean methods were starting to be recognized by software developers as far back as the early 1990s, it was the publication of the Agile Manifesto (www.agilemanifesto.org) in 2001 that defined this new environment where the sole focus is on understanding customer values, and then tailoring everything in the process to produce, deliver and service what the customer wants.

A Change in Attitude

Agile requires culture changes as much as methodology changes. The old command-and-control approach to product development is replaced by collaboration – cross-functional teams of people, made up of all the stakeholders, who are free to use their talent and enthusiasm to work toward a common goal, and where change is embraced as part of the process.

It was time to investigate how we could bring this new attitude to Primavera.

Scrum as the Foundation

Our search led us to Scrum, an Agile process for managing software development projects. In Scrum (the term is taken from the mechanism used in rugby for getting an out-of-play ball back into play), projects are broken down into repeatable, time-boxed iterations that allow for incremental growth of the product and a rapid response to change. It is scalable from single projects to entire organizations. And most important, because Scrum wraps around existing engineering practices, it would bring a new level of flexibility to our project management structure – not be in conflict with it.

It didn't take long to realize that despite the risk of turning our development model upside down, and in spite of the naysayers who said Agile could never work in a project management environment, it was exactly what we needed.

A New Set of Rules

We were ready to start, but how? We were familiar with the work of Ken Schwaber, the co-author of Scrum, through his book Agile Software Development with SCRUM, (Schwaber &Beedle, 2001) and invited Ken to meet with us and give us some guidance on how to initiate Scrum at Primavera. Ken, who ultimately agreed to be our coach. We quickly discovered that our project management processes were not only a good fit with Agile programming, but that Scrum was actually a project management principle that requires discipline and structure -just in a new and creative way.

Scrum has its own set of rules and terminology that enables team empowerment and adaptability. Under Scrum:

• Teams are self-directed and self-organizing;
• Tasks, or mini-projects, are taken from a “release backlog” that is always evolving under the direction of the project owners;
• Each sub-project is begun and completed in pre-defined time increments, typically two to four weeks, or in Scrum terminology, “sprints”;
• Each of those sprints begins with adaptive planning;
• Teams meet daily for a 15-minute stand-up meeting to discuss team-related issues;
• Team progress is updated daily, and displayed using a Sprint burndown chart
• Stakeholders are invited to a demo at the end of each sprint;
• Each sprint delivers tested, fully-functional software, which leads to continuous improvement. Bugs are fixed within each sprint, not at the end of the project;
• Each sprint is fully tested and never more than 30 days from potential release to the customer; and
• Sprints are developed within a sustainable pace, usually within 40-hour work week.

The Start of a New Momentum

Just as we were seriously exploring Agile, we were preparing the launch of a new and very complex software release, one that required the integration of a new workflow and collaboration capability into our current product. Although we were apprehensive about making this kind of dramatic change at such a critical time, we were committed to making it work.

We began our metamorphosis under the guidance of Ken Schwaber, who began by holding a workshop for all our developers to discuss the principles of Agile and Scrum. We then trained and certified 15 ScrumMasters. In Scrum, ScrumMasters are responsible for facilitating the collaboration between the Product Marketing Department and the development teams in order to build the desired functionality within each sprint.

We divided our entire development team into groups of from 7 to 10 people – sprint teams -- and then held the required review at the end of our first 30-day cycle. Because one of the rules of Scrum is that only potentially shippable software can be shown at the end of each sprint, the new functionality presented by the teams had already been tested and documented.

The results generated excitement on every level of the organization. From the development teams to Marketing to company executives, everyone could see that the process worked. In just one month, we had established a starting point for changing the way we develop new products. Development teams were re-energized, Marketing enthusiastically began brainstorming about the possibilities for the future, and a new momentum was set in motion at Primavera.

Adding Test-Driven Development (TDD)

With the fast pace of developing new software functionality across multiple Scrum teams, we realized that we also needed to improve our engineering practices. We sought help from Bob Martin at Object Mentor to help us introduce selected processes of eXtreme Programming (XP). This lead us to embrace the concept of TDD whereby programmers write a test and automate it before they write a single line of code. Once the test requirements have been automated, developers can write a small code segment, test it, write more code and test it again. This continuous process of automated code-checking identifies problems before they can build and create chaos at the end of the project.

By adding TDD, with its focus on software development, to Scrum, with its focus on project management, we could begin to grow our solutions rather than develop them from scratch.

Putting Scrum Into Practice at Primavera

Today at Primavera, there are 11 Scrum teams and 25 Certified ScrumMasters located at our headquarters in Philadelphia and in two offices in India. Almost all of our software is now developed and managed using Primavera project management software in an Agile environment.

Our Agile development processes are tied to the heart of project management, the work breakdown structure (WBS). There is a defined structure and clear areas of ownership within the WBS that provide the framework for implementing Agile processes in our software:

• The top level of the WBS is the function area, This level is owned by the Product Marketing Department, which determines what high-level functional areas they would like the developers to work on for the next release -- for example, “Web project management” or “portfolio management,” or other functions that we may want to deliver with our next release.
• The second level of the WBS contains the desired features, negotiated between Product Marketing and our business analysts. At this level, we may want to add “user-defined fields,” to the portfolio management function area, for example.
• In the third and lowest WBS level, the requirements level, we further break down the features into detailed requirements. For instance, under the feature “user-defined fields,” we may want to be able to “display a list of user-defined fields” or “delete user-defined fields.” This area is managed by our business analysts.
• The level below the WBS are a variety of activities. Each Scrum team -- programmers, designers, testers -- selects the requirements they can accomplish in a sprint. In a detailed planning session, they break the requirements into 8- to 16-hour activities and assign them to themselves. With this a new sprint is underway.

There is a clear sense of ownership at each level of the WBS. What's more, the process allows for an ongoing prioritization of requirements. Product marketing can constantly re-examine its priorities, along with those of our customers, and reprioritize the functional areas and features in the release backlog. Meanwhile, Scrum teams own the details of how the work to deliver the requirements will be accomplished. Stakeholders have the confidence that the highest priority requirements are being implemented. And development is no longer a deterministic process as change is now built-in as part of the process.

Working as one

In a collaborative environment, information-sharing is paramount and requires many levels of communication. All the stakeholders and project participants must be aligned around project goals. Ongoing project status also must be visible to all. Under the rules of Scrum, this is accomplished through a series of regular briefings.

Daily meetings

On a daily basis, the sprint team holds a 15-minute meeting in which each team member relates what he or she did yesterday, what the plan is for the current day, and any obstacles that might be on the horizon. These daily briefings are critical because this is where the team members are held accountable to each other. Remember that in Scrum, management is not telling the teams what do to and how to do it. That responsibility falls to the teams themselves, and it requires a lot of mutual trust and collective ownership.

Each sprint cycle is coordinated by a ScrumMaster, whose job is to remove any obstacles – passing them up to the project manager when necessary -- and clear the way for the team to keep its momentum going forward. (In many Agile environments, there is no need for a project manager. However, we found it helpful to have a project manager to assist the ScrumMasters by removing obstacles at levels beyond their control.)

Using updated project status information, the ScrumMaster generates daily burn-down charts for the team, which are created in Primavera software. Then at the end of each sprint, a ScrumMaster creates a release burn-down chart that provide an up-to-date view of the release status and estimated hours remaining. (See Exhibit 1.)

These burndown charts are critical because they track and graphically show the status of the release. For example, if all the teams involved start to develop a release that contains 120 requirements, and we know that together they have the capacity to satisfy 20 of those requirements, by the end of the first sprint, we will have 100 requirements left. At the end of the second sprint, there will be 80 requirements left, and so on. As we “burn down” the release, the burndown chart documents that progress and keeps everyone aware of the project status. As the team completes each requirement, Primavera software marks the corresponding WBS element as being complete in the release view.

Exhibit 1 – Measuring Progress: A Sprint Burndown Chart

Weekly meetings

The ScrumMasters meet weekly to discuss and resolve cross-team issues, such as how to resolve development goals overlap on a particular project.

Sprint Review

At the end of each sprint, the team demonstrates its work to all the stakeholders – Product Marketing, company executives and key customers – and asks for feedback. We also provide our key customers a WebEx web conference at the end of each sprint so that they may give additional comments or requests to Product Marketing, which in turn gains additional guidance for resetting their priorities and determining which items should be moved to the top of the release backlog.

The Scrum team then returns to the reprioritized release backlog to pick off the requirements they will work on during the next Spring.

Measuring Progress

Although quality metrics do not change with Agile, the metrics for tracking projects in Agile are different from those of traditional project management. For example, under Agile, project metrics are generally team-based, rather than individual-based, and the goal is to continuously pick up speed and improve. Therefore, the primary thing that we measure is the velocity of the team. We know how many units of work, or requirements in our case, the team can do in a sprint. We can use progress information to forecast how many requirements can be completed in the next sprint based on the requirement's weighting. And when we combine the work across teams and their velocities, we can predict how long the new product will take to complete.

Agile and Project Management: A Success Story

There is no doubt about our success with incorporating Agile into our traditional project management processes. It has allowed us to significantly improve the quality of our software. Within the first nine months of releasing our first Agile-developed product, we saw a 30 percent increase in quality as measured by the number of customer-reported defects.

We also experienced faster time to market. By using Scrum, Product Marketing was able to adjust to market conditions and consolidate two planned releases into one backlog containing the highest-value items from both releases. The result was that we delivered the combined release four months ahead of schedule. Before Agile, we would not have been able to make this kind of change midstream and deliver one release with a combined feature set.

There were also benefits to our development teams. Agile allowed us to create a long-term, sustainable work pace for the sprint teams, and gave team members the opportunity to grow and work outside their roles.

The new atmosphere of teamwork helped the teams build trust among their members, which is fuel for high-performance teams, and because they had full ownership of their work, they could take pride in themselves as they demonstrated their accomplishments to the stakeholders.

At the completion of a project, the developers were refreshed and enthusiastic and ready to start their next collaboration.

Finally, the benefits to Primavera as a company have been incalculable. Stakeholders are now involved throughout the development process and can give immediate feedback. Our Marketing Department is able to concentrate on the highest-value features and make appropriate changes on a monthly basis. Management has a new regard for the talent and commitment of its employees, and the developers are enjoying an atmosphere of respect and continuous learning.

What We've Learned

Energized by our success with developing our solutions in the Agile method, one of our goals is to help others achieve the same efficiencies that we have enjoyed and, ultimately, to add new value to their organizations.

For those who may be considering making the move to an Agile environment, we want to share our experience and make your transition a little smoother. Here's what we learned:

• Seek a sponsor, a champion who is committed to making it happen. Your sponsor has to be willing to stand up to the critics, encourage the leaders and effectively communicate the vision of the team.
• Use outside coaching. You will need honest and objective feedback from an outside source – when you're insolated, you don't know what you don't know. At Primavera, coaching helped us enforce a learning culture. We were able to learn new techniques in iterations, try them, and see where we could improve and gain confidence as we progressed. But most important, we learned not only to accept change, but also to initiate change as part of our new culture. In an Agile environment, most of the change comes from the people doing the everyday work, not from management. Once you have that culture in your organization, things start to move very smoothly.
• Focus on teamwork. Some people find teamwork a natural way to work. But for others, having to share their knowledge can cause anxiety and resentment. So it's important to put a significant focus on team dynamics and the personal satisfaction that can come from being part of a self-managed team.

  People working together doesn't make a team. Getting people to work with unity of purpose takes time and attention. One of the key points of Agile is that work must be conducted at a sustainable pace – which means the elimination of the peaks and valleys that might lead to working unusually long hours -- so part of team building is also learning the discipline of developing software at a sustainable pace. Again, this takes time to work out.

  Team building also requires management to look at their work in new way as they learn to shift their focus from assigning tasks to delegating some of their authority to the self-managed teams.

  This requires exceptional project management tools and solid engineering practices. The lack of good checks and balances, good processes and good project management can lead to chaos as people learn to work within this new culture. With the backbone provided by solid project management and engineering processes, people have a structure that frees them to make changes and improvements that build on those established principles.

• Use the established Agile language. Agile has its own terminology and embracing it is a critical part of moving to the new culture. Using the new language of Agile encourages people to think in new and creative ways.
• Get executive support. Because Agile is driven from the bottom-up, it requires complete support at the executive level. As people learn how to work in this different environment, there will be new challenges to meet and mistakes will be made. Having executive support frees them to try new things and continue to grow.
• Expect hard work. Saying you are Agile doesn't make it so. Moving from the traditional waterfall approach to an Agile culture, means turning an organization upside down. There is no silver bullet – it will take commitment and hard work.

The Road Ahead: The Future Looks Bright

There is no reason why the tenets of project management – earned value, resource management, time management, etc. -- should be considered inconsistent with Agile. In fact, we think they are perfect together. We've been working with Agile development for three years now, and we see unlimited opportunity ahead as we continue to blend Agile development with our project management software.

For instance, one of the things we would like to utilize within Agile is the use of earned value. With a better understanding of how much work we actually accomplished compared with how much work we expected to accomplish, we could more exactly predict how many sprints will be required to complete a project. One of the keys to the sprint process is estimating, and earned value is the perfect tool to make it more accurate. It would allow us to review our estimates on a sprint-by-sprint basis, and to better determine how well we are able to deliver what we promised.

Going forward, we also want to use our own software to do more in-depth resource planning for the sprints. At the beginning of the release, once the Scrum teams have been identified, they easily map to resource teams within our software which can then facilitate skills matching. Further, by doing resource leveling at the WBS level, we could more easily forecast what we can complete and when.

These are the kinds of things that will help us continue to improve our processes, and they are just the beginning of our journey down the path of totally integrating traditional project management with Agile software development within Primavera.

Agile is a commitment to collaboration, empowerment, respect for one another, courage, flexibility, honesty and trust. It has given Primavera an energized workplace and a new potential for creating even better products and services. We are excited about the fact that other organizations are now asking us how they, too, can gain a new momentum through Agile techniques. And, we are proud to be recognized as industry leaders who continue to embrace new and innovative methods that bring us closer to our goal of eliminating project failure.