Planning for productivity measurement in an application outsourcing contract

"value in the details!"

Introduction

You are handed an applications outsourcing contract exhibit with a single sentence about the contractual obligation to demonstrate productivity improvement over the life of the contract. Your responsibility is to be the project manager (PM) for the project to establish the Productivity Baseline and processes for on-going measurement and evaluation. The questions you should be forming in your mind would include:

  • How will you measure productivity?
  • What types of work are being measured?
  • What kind of activities are part of this project?
  • How long do you think it will take?
  • What kind of problems you need to plan for?
  • What are the success factors?
  • What skills are needed to accomplish this work?

If you have never constructed a viable approach to this productivity obligation, the assignment could seem daunting.

The objective of this paper is to provide an approach and a sample plan to establish a productivity baseline of services for an Application Outsourcing Services (AOS) contract and a process for on-going measurement and reporting of performance in comparison to the baseline.

The paper presents an approach that incorporates both the client's view and the service provider's view of productivity in an outsourcing agreement. The approach includes data gathering and process design activities as well as methods for managing productivity improvement based on quantitative analyses. A sample high-level project plan will address productivity definitions, data availability, client's expectations, data analysis techniques, and the potential actions that might be taken in response to actual productivity measurement results.

What is Productivity?

There are a number of ways to define productivity. For example, a Google search of the internet returns over 5,650,000 references. Some definitions seemed better than others, but the best one found was “Productivity is the Ratio of output to input” (NPCC, 2005).

In other words, productivity relates a measure of output to a measure of input. It is easy to select a measure of input like effort, duration, price, and cost, to mention a few. This is typically directly tied to the objective of the baseline. For example, the client is interested in measuring cost saving due to outsourcing. Since a major component of the cost in AOS is labor cost, a baseline of technical productivity (i.e. units of output per effort hour, or person month) is required, and, therefore, effort is the measure of interest. Duration would be a measure of choice for schedule productivity, when timeliness is the characteristic of concern.

Determining measures of output is more challenging. The size of a product produced or supported should be a natural measure of output. However, in the business of applications work, determining the size of application output can be difficult. Function Point (FP) is a good sizing measure, as it is technology and language independent. One drawback is that it is often prohibitively expensive to baseline a large application portfolio in function points. The choice of application/project size is contract specific and needs to be resolved within the contract context and constraints: data availability, cost, timeliness, etc. Advantages and limitations of different size metrics are important within a baseline context, but are outside of the scope of this paper. The examples in this paper use FP for New Development and Enhancement Projects and Source Lines of Code (SLOC) for maintenance and production support. The process can easily be generalized to calculate productivity for any type of work, (e.g. fulfill report requests) for which size and effort can be collected.

It is assumed that the work within the Application Outsourcing Services (AOS) contract is performed through projects that follow a formal Project Management discipline, and therefore productivity of individual projects can be calculated if units of output and input are chosen. Effort is the choice of input in this paper, and a unit of output (size) depends on the project type and will be addressed below.

Therefore,

Productivity = Output / Input

or

Productivity = Size / Effort.

But does this formula imply that it takes the same effort to develop/maintain a unit of size? It is well known that there are a number of factors that impact effort (Jones, 2000; Bohem, 2000; Putnam, 1996,) and therefore productivity, of the same size projects. To ensure fair comparisons of productivities, we developed a method of classifying projects into “like” categories as a part of Productivity Measurement Process. EDS calls this tool the “Output Based Delivery Model” (OBDM) and it will be briefly discussed later in the paper.

Productivity Measurement at a Glance

Exhibit 1 presents an overall view of the Productivity Measurement Process. It provides a framework for planning a productivity measurement project by identifying the key process steps

Productivity Measurement Process

Exhibit 1. Productivity Measurement Process

This is an iterative process that provides creation of the productivity baseline, ongoing productivity measurement, reporting, and continuous improvement. It facilitates cause and effect analysis for productivity improvement. The process presented here requires a set of clear definitions to ensure shared understanding of terms which are documented in the following process steps.

Steps to Establish Productivity Measures

Define Scope / Approach / Model

A Productivity Baseline is a snap shot of performance at a specific time, where Productivity is chosen as the measure of performance. It could be compared to a start line, which identifies a capability of the system to compare to in the future.

Like all projects, the most important step in implementing productivity measures is to establish scope and requirements. Consider bringing the stakeholders together for requirements gathering / project planning meetings in order to bring the proper focus on the subject. The primary decision makers are often deeply involved in other aspects of implementing the contract. The discussions and decisions reach down into the details of how each company views the work being measured and can become complex and confusing. A properly run session will save time for all stakeholders and make sure the project gets off to a good start.

Creating a baseline scope requires identifying types of work to be included and a time interval (historical or going forward or combination of both) that will be the source of data. The client and service provider must agree on the output metric for each type of work (size), the metric for input (effort, FTE, $, etc.), intervals for measurement, how to compute overall productivity, and reporting formats. It is also important to address data availability, ownership of data, and data security. Some of these items will be defined in the AOS contract. However, the contract language may be vague (intentionally or unintentionally) so make sure these are defined at the beginning of the project.

The following aspects of baseline scope need to be addressed with special care to provide a sound foundation and approval for the successful Productivity Baseline project:

  • A representative sample of work to account for the work mixture in the project portfolio that will be tracked going forward;
  • A sufficient sample size that supports a valid statistical analysis of data;
  • The metrics used in the baseline data and for on-going productivity measurement must be consistent with each other.

A common pitfall is that effort in the baseline is for work different than that called for in the contract. For example, in case of a historical baseline, the client would have done all of the work – from requirements through implementation. Under the AOS contract the client may retain some of the work and give the rest to the service provider. The service provider may not be performing all of the life cycle tasks that were done by the client so the productivity looks higher, but in reality the actual productivity change is unknown. In this case the effort for the client (historical) work needs to be adjusted to include in the baseline effort only the same tasks that will be performed by the service provider.

Make sure you know the definitions of the metrics and how they relate to the AOS contract. AOS contracts often include actions to be taken for exceeding or missing productivity targets. The types of actions are contract-specific and won't be discussed as part of this paper, but the process must be designed to provide the data to implement the required actions.

As mentioned in the “What is Productivity” section above, the following section provides general ideas of the development of a classification model that accounts for different complexities of projects within the portfolio of applications.

Output Based Delivery Model (OBDM)

The following is a list of considerations that went into the development of the model:

  1. Every AOS project either produces new/added software code (New Development and Enhancement Projects) or supports existing software products (Maintenance and Production support projects). New Development, Enhancement, and Maintenance are three work types. Each work type requires separate baseline.
  2. Effort is a major cost in application development and maintenance (ADM) work, and factors that relate cost to size (Jones, 2000; Bohem, 2000; Putnam, 1996) make a good set of attributes to define “alike” projects.
  3. The mixture of work may change over time. Statistically representative samples of projects/applications within a large corporate portfolio are required to account for the difference in work and to calibrate the model by establishing boundaries of parameters that define categories of “alike” projects.
  4. All factors in a cost model may be divided into two groups: Business Factors and Productivity Factors. Business factors are controlled by the Client. Productivity Factors are controlled by the Service Provider. Project/Application Size, Application Criticality, and Reliability Requirements are examples of Business Factors. Productivity Factors examples are Product Characteristics, Project Structure, and Team Skills.

OBDM is a two-dimensional model that assigns two numerical values (CI, PI) to every project, where

  • CI – Composite Index: derived from the set of Business Factors, and
  • PI – Productivity Index derived from the set of Productivity factors

Based on the distribution of (PI, CI), the OBDM model uses percentiles to classify all work within a work type into three categories (Bands) being ”Easy”, “Average” and “Difficult”. As a result of the model calibration, every project/application is marked as “Easy”, “Average”, or “Difficult” as illustrated in Exhibit 2.

Output-Based Delivery Model Classifications

Exhibit 2. Output-Based Delivery Model Classifications

The Normalized Index is a set of three multipliers NI = {NI Easy, NI Average, NI Difficult}
that relate units of work across the bands to the complexity of work in Average band, as follows

  • Unit of Work (Easy) = NI Easy * Unit of Work (Average), NI Easy<1
  • Unit of Work (Average) = NI Average * Unit of Work (Average), NI Average = 1
  • Unit of Work (Difficult) = NI Difficult * Unit of Work (Average), NI Difficult >1

For the sake of brevity and simplicity, we will not go in the details of the algorithm. Whatever model you choose will impact the final presentation of the baseline, but we believe that the general process and project structure will be common.

Other Benefits of an Output-Based Delivery Model

An OBDM provides more than just a way to account for the factors which impact productivity while measuring productivity change. Once the service provider builds a historical view of actual project performance, the data can be used as an aid to estimating. Projects of similar size and complexity will belong to the same category, and can be expected to have similar productivity. Average productivity within the category provides a good measure of expected productivity of similar projects, so the model can be used to validate the estimates produced by the project team. It also can be used to identify areas where efforts to improve productivity are needed most. It provides the most value by allowing analysis of the contribution of the individual factors included in OBDM and controlled by the service provider.

Set Rules to Establish Baseline

The client and service provider productivity teams work together to design the processes. They must ensure the processes are understood by both parties and accomplish the goals of productivity measurement. One major goal for this activity is to create a process that is transparent and auditable. Another goal is to minimize the effort required to measure productivity.

Productivity measurement will be integrated into the overall metrics program for the contract. All productivity data and calculations must be traceable back to the primitive metrics (i.e. size, effort, defects, etc.) Collection of primitive metrics should be integrated into the work activities in order to minimize the amount of time spent collecting productivity data. As mentioned in the previous section, analyze the primitive metrics data to ensure that only data that is consistent with baseline data is brought into the calculations. This analysis should be done early in the life of the contract so that the appropriate data can be collected while work is being done. At the latest, it should be done as part of requirements gathering / project planning.

Collect Baseline Data

As stated above, there are two main choices in the selection of the baseline(s) – historical and current.

  • Historical - measures productivity change from past work done by the client
  • Current - measures productivity change from the current service provider's work.

In either case, the set of projects and applications to be included in the baseline must be identified. A historical baseline requires availability of client's data and willingness to share it. If the client has existing metrics repositories for size and effort this may be as simple as feeding the data into the model, and producing reports that calculate the baseline.

If metrics data does not exist, sub-projects may be needed to establish the data. e.g. perform enhancement sizing (FP counts) for the baseline projects or size (count SLOC) for baseline applications. These activities can require significant time, tools, skills, and effort so a validation of the cost / benefit analysis is warranted before starting.

The OBDM requires information about the Business and Productivity Factors. For our purposes these factors were collected via the AFOTEC (AFOTEC, 1996) and COCOMO II (Bohem, 2000) surveys – although other methods can be used. Each of the questions within the surveys must be categorized as a business factor (controlled by the client) or productivity factor (controlled by the service provider.) This is another area where a complete understanding of the AOS contract and the roles and responsibilities of the client and service provider is required.

Calibrate Model

This part of the project calculates parameters of the OBDM from the baseline data using OBDM algorithm. The following are steps of the model calibration

  • Calculate boundaries of (CI, PI) for each band.
  • Compute values of Normalized Index NI ={ NI Easy, NI Average, NI Difficult}, NI Average = 1
  • Individual baseline projects are classified into the appropriate bands of Easy, Average and Difficult based on the values of CI, PI.
  • The expected productivity of each band is calculated as average productivity within the band.
  • Individual work units are then converted to equivalent work units by applying the corresponding NI Multiplier.
  • Overall Baseline Productivity for each work type is computed as a ratio of the total sum of equivalent work units across all bands to the total effort.

Future projects that are done by the service provider will be classified as Easy, Average or Difficult using (CI, PI) boundaries defined during the Model Calibration. Before the productivity change vs. the baseline is calculated, the productivity for individual projects will be calculated using equivalent units of work (input) for the band.

Create Reports

Baseline productivity for each type of work (New Development, Enhancement, and Maintenance) is calculated according to the rules established during the requirements phase. The baseline reports are validated by the client and service provider (the amount of agreement/disagreement you encounter at this time is directly proportional to how well you were able to establish clear requirements.) Some of the validation activities could include third-party review of FP counts, review of sizing processes and tools and a detailed inspection of the OBDM model. This is also a good time to create simulations of future project performance in order to fully validate and understand how the productivity model and productivity change calculations work.

Measure Productivity Improvement

Collect Productivity Data

At the agreed-to intervals (e.g. quarterly, semi-annually, or annually) the productivity data is pulled from the metrics repositories. The OBDM band is assigned to each project based on the values of CI and PI of the project and band boundaries established by Baseline Calibration Step. Project size is converted to equivalent work units using the Normalized Index value for the band to calculate productivity. The productivity change from the baseline is then calculated and summarized for each baseline (recall the separate baselines are calculated for each type of work.)

Create Reports

Each time a productivity report is created the client and service provider will review the input data and validate that it was collected and reported correctly. Once agreement is reached on the reports the appropriate actions can be taken based on the change in productivity.

Analyze Productivity Data

As mentioned previously in this paper, the OBDM provides ways of looking at productivity that help identify the best place to initiate productivity improvement projects. For example:

  • Analysis of productivity variability of individual projects within each band to identify best practices of projects with higher productivity;
  • Analysis of service provider controlled factors that may improve product maintainability
  • Review change in the mixture of work;
  • Analyze projects by platform or language across the bands to see if process changes or skills development are required.

Maintain the Portfolio

An application portfolio changes over time. New applications are created and old applications are retired. When new applications are added, the productivity baseline for that application must be established and an agreed-to period of time to track productivity before measurement of productivity change can take place. The parameters for this can be established during the requirements phase or they can be determined for each application as it is added to the portfolio. A typical scenario is that the baseline productivity is established after the first 6 months that the service provider is responsible for the application and that productivity change is measured after the next 3 months.

When applications are retired the data associated with the application is removed from the baseline once the final productivity for the application has been reported.

It is also important to analyze change in the portfolio to validate relevance of the original baseline. It is done by recalibrating the model and comparing the new model parameters to the original baseline model. The simplest step to identify change in portfolio is to compare new ranges (Min CI, Max CI) and (Min PI, Max PI) to the corresponding original baseline ranges. This activity is simplified by using a comprehensive model calibration tool that automates data consolidation and calculates the parameters.

Summary

Measuring productivity change is becoming more common in Application Outsourcing Services (AOS) contracts. While the technical definition of productivity is simple, there are many pitfalls, issues and complexities that must be addressed before a fair productivity measurement process can be established. The paper has identified several critical success factors:

  • Well-defined scope and requirements;
  • Transparent and traceable process;
  • Reports that support evaluation of whether or not productivity targets were achieved;
  • Reports that help identify areas where productivity improvements will give the best return on investment.

An Output-Based Delivery Model (OBDM) can be used to account for the differences in project productivity that are caused by productivity factors. Using an OBDM gives a more accurate measure of the expected productivity for a project. In addition to its value in the calculation of productivity change, an OBDM can help with estimating processes and identifying areas for productivity improvements…the value is in the details.

References

AFOTEC (1996) Software Maintainability Evaluation Guide, AFOTEC Pamphlet 99-102, Vol. 3

Bohem, B., Horowitz, E., Madchy, R. Reifer, D., Clark, B.K., Steece, B., Brown, A.W., Chulani, S, & Abts, C. (2000) Software Cost Estimating with COCOMO II, Upper Saddle River, NJ: Prentice-Hall

Jones, C. (2000) Software Assessment, Benchmarks, and Best Practices, Boston, MA: Addison-Wesley

National Productivity and Competitiveness Counsel (2005) Productivity Definition, Retrieved on 07/05/2005 from http://www.npccmauritius.com/definition/

Putnam, L, Myers, W. (1992) Measure of Excellence: Reliable Software on Time, within Budget. Upper Saddle River, NJ: Prentice-Hall

© 2005, Olga Makar-Limanov, Tom Stratman
Originally published as a part of 2005 PMI Global Congress Proceedings – Toronto, Canada

Advertisement

Advertisement

Related Content

  • PMI Sponsored Research

    Digital Transformations of Traditional PBOs and Modern PNWs member content open

    By Braun, Timo | Ekstedt, Eskil | Lundin, Rolf A. | Sydow, Jörg Digitalization is a phenomenon occurring across sectors and nations, affecting technical processes, organizational forms, and managerial practices. Project management, which is often used as an…

  • PM Network

    Banish Distractions member content open

    By Husted, Chad If I have learned anything in my 20-plus years of project management, it's that there is always room to grow and improve my skills. Yet, some of the best lessons I've learned about being an…

  • PM Network

    Standing out from the Start member content open

    By Scott, Lindsay Project management recruitment professional Lindsay Scott provides career guidance.

  • PM Network

    Ready, Set, Stretch member content open

    By Bishel, Ashley A strategic skill set can guarantee a static career. But for project professionals eager to be recognized, get promoted or take on more ambitious and complex projects, an expansive skill set is a…

  • PM Network

    Exit Strategy member content open

    Uncertainty looms for project professionals in the United Kingdom. But there's hope amid the Brexit chaos: Salaries are up—especially for those who change jobs.

Advertisement