Project Management Institute

Customizing system testing discipline for agile


Over the past three decades information technology has gone through an evolutionary process, which has helped in changing the fragile, disjointed, and buggy applications into robust, monolithic and ever encompassing solutions for practically all the process areas that one can contemplate. In today's world, the role of developers is limited to either maintaining these applications or converting them to a newer platform and in the process adding some additional features that improve the functioning of the business process. The agile methodology is perfectly appropriate for these kinds of application management. System testing is an essential part of any application development but is sometimes misconstrued by the application managers, which leads to diluting the testing efforts for the sprints. Many organizations are trying to redefine the testing strategy for agile development and we realize that integration of tools plays a big role in this initiative.

The author has used different tools and processes in developing a testing framework for the agile methodology and wants to share his experience in this article. The author used the Testing Process Maturity Model (TPM) to assess the organizational maturity to manage the QA/QC activities in the application release process. A few projects were used as pilots to identify the critical areas in implementing the user stories, using the agile project management tool, Mingle. The main objective was to use the traditional techniques and fine tune them to match the fast pace of agile without compromising the essential validation/verification requirements of the application. A new concept of “developer–tester” was used to introduce the “pair programming” and “test-driven development” approaches of agile.


In today's scenario we rarely see any big application being developed from scratch. Pretty much all business functions are now available as products in the market, where you need to customize the available products to suit your requirements. Every product supplier has a process defined for customization to make the change less painful and quick. The “buy” decision has become a default option where we get into a “make or buy” decision for any computation needs. So, if we ask what kind of activity today's developers are engaged in, the answer is that, basically, they are doing either “enhancement maintenance” or they are busy adding new functions to meet business and regulatory requirements. Usually, bigger projects are conversion projects in which the developers migrate an existing legacy application to a more modern technology. These projects are also done in incremental fashion and not according to the “Big Bang” model. So, essentially, we find that today, and for years to come, developers will be working on smaller sizes projects rather than huge, multi-year projects.

Whether it is a waterfall or a RAD, the need for verification and validation cannot be eliminated. In the current scenario, we expect the development to happen at a much faster pace. So, the challenge is how we can provide the V&V service without slowing down the process but at the same time not compromising the quality. In this paper, use of the Agile Testing Framework, which requires the appropriate tools to meeting that challenge, is discussed.

Why Agile?

As discussed above, at present, application development is confined to smaller, incremental efforts. The other phenomenon observed these days is that business users are becoming more and more technologically savvy. It is not uncommon to find a business user creating his or her own little application on MS Access or on Excel, with macros to meet his or her specialized needs. This group of users is knowledgeable about the data structure and the application architecture. This is a huge advantage in collaborative programming, which benefits both the business community and the developers.

There are several reasons why agile is becoming a de-facto practice in the industry: One of the biggest reasons, of course, is risk mitigation. Traditionally, the waterfall method development takes a year or so to implement and we often find that the requirements change during this duration and the applicability of the system lose their relevance. This leads to a waste of time, money, and effort on the part of the company. Since agile is incremental and developed in quantum, it outpaces the change that is happening in the environment; thus, it mitigates the risk to a great extent.

The other major reason for agile is “time to market.” The core function is delivered first so that the basic requirements of the function are available to the users to start conducting business, while the other supporting functions are delivered through subsequent sprints.

This approach also provides opportunity to business users to tweak the new product strategy much faster based on market reaction.

Organizational Maturity

If we look at history, we find that Van Gogh replicated paintings of famous masters from the early Renaissance period before he adopted his bold impressionist style. Similarly, every organization needs to go through the graduation steps by refining its development process through traditional methods before adopting agile. Agile does not mean “not doing the documentation” or “not performing verification.” Rather, it is a very sophisticated process through which one can deliver the functions in an incremental way with higher levels of accuracy and perfection.

The TPM model is a good way to measure the organizational maturity in dealing with system testing. This model uses 20 factors, which are measured between 1 and 5, to determine the level of the organization. Exhibit 1 illustrates the spider diagram, which is the outcome of this analysis.

Spider diagram of TPM

Exhibit 1 – Spider diagram of TPM.

There are four levels of maturity as defined by this model: Starting, Controlled, Efficient, and Optimizing. A maximum achievable limit is defined for each factor. The model has detailed level sections that assess the organization's capability with reference to those 20 factors through a set of questionnaires. The red line is the result of this assessment and it shows the maturity of the organization against the highest level that can be achieved in that factor. For example, the highest achievable level for “Life Cycle Model” is 4 and in this picture it shows that the organization has achieved a level of 1. Similarly, in the case of “Test Strategy,” the maximum achievable level is 12 and the organization has achieved a level of 4. This way, the diagram provides an overall picture of what needs to be done to improve the standard against industry practice.

Test Plan

Test Plan creation is very critical for the agile model. Typically, in the waterfall model a Test Plan is a 20-page document. To make it suitable for agile, the following Test Plan (Exhibit 2) is designed, which is a condensed version of that detailed plan. This Test Plan is defined for a module that can be completed in one sprint.

Test plan

Exhibit 2 – Test plan.

Test Strategy

The Test Strategy section identifies the types of testing required for the sprint. For each type of test, the responsible department (i.e., Business User, Production, or the QA Testing) is identified in the “Who” section. What function or code is going to be tested is defined in the “What” section. The mode of testing like manual or automated testing or testing with test harness is defined in the “How” section. If a certain type of test is not planned, then a justification is stated for it not being performed. In this template, some of the standard test types are specified but more test types can be added in specific cases, as per the testing strategy. Risk-Based Testing is also part of the test strategy where the risk for not performing a certain type of test or not performing certain Test Cases is detailed. In reality, quite often we are under lot of time pressure to release a specific function to meet the business demand. This type of situation is most suitable for Risk-Based Testing. It is a good idea to do an FMEA analysis to identify the most vulnerable areas that are susceptible to failure.

Effort Estimation

The next section of the Test Plan is allocated for effort estimation. Some of the tests are duration based like Performance testing, whereas some of them are based on the volume of test cases that is required to be run for the sprint. This may be contradictory to the agile philosophy, because we do the estimate in a relative sense and not in absolute terms (like person hours). In agile, the effort requirement is not definite as the backlog is elaborated and as we progress through the sprint. On the contrary, in the case of testing, activities are very well quantified; for example, we know how many test cases will be executed as part of Regression Testing.

Test Environment

The following section defines the requirement of Test Environment. Preparation of Test Environment is a critical activity, because the success of testing depends on completeness of all the components of the application. Maintaining the test environment and keeping it updated with the constant changes is a challenging task. Selection and preparation of environment are also dependent on the test data availability. In some cases, it is required to prepare the test data to successfully test the functions that are newly implemented through the sprint. In agile, because we develop the application incrementally, more often than not, the “calling function” or the “called function” does not exist in the environment. In those cases, a Test Harness or a Test Stub must be developed to invoke the test case.

Test Automation Plan

Test automation is a very important aspect for agile environment because it helps in maintaining the fast pace of the development process. It requires a “detail level study” to identify the test cases that qualify for automation. For a test case to be executed in an automated manner requires effort from a tools perspective and also from a data perspective. In this section, the requirements for test automation are planned.

Risk Assessment

Risk assessment in any kind of planning is a good practice that tries to identify the possibility of something going wrong and its impact on the program. Non-availability of certain types of information poses a risk for the program, and in this section the risk involved for such situations is identified and mitigation strategy is defined.

Tools Usage

Tools play a very important role in maintaining the agility of the agile model. Tools help in automating the testing operation and also help in keeping the testing record and traceability of activities performed.

Selection of Tools

There are various tools in the market that can be used in the testing process, which range from requirement gathering, project management, test-ware management, test automation, defect management, configuration management, and deployment to test environment creation. The most important thing is that all the different tools need to interface with each other to share information. While selecting the tool, we have to make sure it can perform the function that we are looking for, but it is just as important to check its capability of exchange data with other tools. Some of the tools provide APIs to interface with other tools, whereas for others we may have to write a utility to make them “talk” to each other. The basic objective is to reduce the duplication of efforts because we cannot afford to increase the workload in the agile model.

Tools Integration

Although we take enough care in selecting the right tools with the right interfaces, we end up with some tools that are fantastic in their own functions but very poor in sharing the data. Exhibit 3 illustrates how an organization acquires different incongruent tools and then ends up putting in a lot of effort to try to integrate them.

Test Management

HP's Quality Center (QC) is a great tool in managing the test operation but it is not a good tool for writing requirement specification. IBM's RequisitePro is probably the best tool on the market for requirement specification, especially, for an object-oriented model and the good thing about it is that it interfaces with QC seamlessly. QC has the capability of linking the different modules very efficiently and it maintains the traceability of requirements to test cases and then to the defects. It helps in managing the testing activities throughout the testing process.

Project Management

There are quite a few tools for project management on the market but the project management tools that support the agile model are still in the initial stages. Mingle is a relatively good tool that covers aspects of the agile model. It is based on a card system in which you can create a card for each story to keep track of backlogs. In the same way, cards can be created for test cases and for defects and to correlate the testing activity with development. Interfacing Mingle with QC is not an easy task but it is possible.

Integration of diverse tools

Exhibit 3– Integration of diverse tools.

Defect Management

Managing defects is the key in the testing process. Testing is all about identifying defects in the application and how we get rid of them before deploying the application in production. An efficient Defect Management helps the organization in not only quick tracking and resolution of defects but also in the isolation and identification of potential process improvement through defect analysis. It is very important to use a suitable tool for defect management to maintain the history of the defects, which in turn enables identification of recurrence of defects. QC's defect management module is very good for the test cycle management. Remedy is a much more versatile tool and takes care of all aspects of service management.

Agile System Testing Framework Model

The typical framework of the waterfall model requires a mindset change to adapt to the agile model. Exhibit 4 depicts the framework from the time when the developer gets the module ready (the blue circle) to final deployment in production (the red circle). In between, the module goes through various rigorous tests, which make it defect free and suitable for business use. This diagram also defines the roles of Developers, Business Analysts, Testers, Business Users, Production, and the Project Managers throughout this process. The most significant difference between waterfall and agile is in the very first section where the Developers are working on the coding of the module. The concept of Pair Programming or Test-Driven Development changes the way development is done. Involvement of Business Users is extremely important in the agile model. They take active part in requirement definition but their job does not end there; they also take part in the definition of system requirement. We expect the Business Users to be tech savvy to a certain extent so that they can help in translating the business functions to system modules. They are also required to define the datasets that are relevant for the new functions implemented in the sprint.

Agile testing framework

Exhibit 4 – Agile testing framework.

The Role of Developer–Tester

This is where the agile model differs most from the traditional waterfall model. For some of us, it is very difficult to perceive that QA Testers are required to perform Unit Testing. Agile brings a new skillset, “Developer–Tester,” to make the development process fast and efficient. While we say the Developers are the “Creators” and the Testers are the “Hackers” or “Destructors,” in the case of agile we need a personality that is a combination of both. We need an individual who is extremely good in programming but at the same time understands testing methods and has a critiquing attitude, which of course, is very difficult to find. The Developer–Testers work very closely with the Developers and, while the Developers are busy coding the module, the Developer–Testers are busy identifying the test cases that need to be included in the list of unit testing. They work with parallel version of the code to incorporate the instrumentation, which will define the pass/fail condition.

The Developers’ Test Bed

We are familiar with the concept of a Regression Test Bed as shown in Exhibit 5. The Developer's test bed is pretty similar to this configuration and management perspective. We need the Developer–Testers to constantly identify the new test cases in the Unit Test environment that will be included in the inventory. These test cases will be automatically executed as a new build is dropped in the Unit Test environment. This mechanism helps the Developers in constructing an efficient code and makes the development a fast-paced process.

Regression test bed

Exhibit 5 – Regression test bed.


The agile Testing Model becomes successful when tools, people with right skills, and the methodology come together. We also have to take into consideration the continuous improvement to Sprint, which happens when we implement the experience gained from each Sprint to the subsequent ones.

This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI or any listed author.

© 2012, Neelov Kar
Originally published as a part of 2012 PMI Global Congress Proceedings – Vancouver, Canada



Related Content

  • PMI White Papers

    How Agile are Companies in Germany?

    By PMI Cologne Chapter Until recently, Agile was considered as a set of principles and practices relevant only to software development projects. However, Agile is now spreading to other parts and types of organisations,…

  • PM Network

    Agile Capacity

    By Parsi, Novid Wrong resources? Right resources at the wrong time? Both can cripple project momentum—and send shock waves across the project portfolio, even threatening the organization's bottom line. And the…

  • White-Paper-Cologne-thumbnail.jpg

    How Agile are Companies in Germany?

    By PMI Cologne Chapter Until recently, Agile was considered as a set of principles and practices relevant only to software development projects. However, Agile is now spreading to other parts and types of organisations,…

  • Design Thinking to Improve Your Agile Process

    By Vukosav, Denis Agile project teams interact with users and deliver incrementally. This paper is an introduction to how design thinking, combined with agile, will further reduce the risk of failing.

  • PM Network

    Practical Guidance

    By Fewell, Jesse The debate about processes—when to follow them versus when to bend them in favor of deeper principles—has been an age-old, fundamental tension in project management. Those of us contributing to A…