Management of 3rd party software development suppliers

Introduction

Many companies are outsourcing software development projects to third party suppliers rather than taking the risk of running the projects using solely in-house resources. This strategy brings with it its own set of challenges, advantages and disadvantages, as well as risks. In order to effectively manage third party software development suppliers, a number of principles and processes need to be applied by the customer company to increase the probability of project success. This paper discusses these principles and processes to assist customer companies embarking on third party software development projects. The content has been verified as adding value, as the author has personally used these methods in effectively managing third party suppliers.

Benefits of Third Party Arrangements

Companies may opt for developing and maintaining all their software systems using internal staff, outsourcing development to one or many 3^rd party suppliers, or a combination of both strategies. The decision will typically be driven by factors such as the availability of specialist skills, costs, on-going support and risk.

One of the major perceived benefits of using a 3^rd party for software development projects is the reduction of risk of project failure and cost overrun. The customer company attempts to transfer the project risk to the 3^rd party supplier, especially through the establishment of a fixed-price contract for the project. The supplier quotes for the project, stating a fixed cost to deliver the required functionality, as specified by the customer, by a specified completion date. The contract may also contain bonus payments for early delivery, as well as penalty payments for late delivery.

Another benefit of using a supplier may be that the supplier has specialised knowledge of the specific application that needs to be developed or of the technology that will be used for the development, which the customer does not have in-house. It is always more risky to use new technologies than using existing technologies known by the customer, so this could be a way of reducing the risk of failure of the project.

Many companies are more skilled at maintaining and enhancing existing systems than doing large scale development projects, which require specific skills such as project management skills. By engaging a supplier to be responsible for the development project, the customer company benefits from the supplier's skills in running similar large scale development projects. This should ensure effective project scoping, planning, monitoring and execution.

In-house systems resources are often split between maintenance work, production problem fixes and projects. This causes prioritisation problems, as the resources are not dedicated to the project work. An outsource supplier, on the other hand, can allocate dedicated project team members to the project, which enables these resources to focus purely on the project. This enables more reliable resource planning and estimating.

Disadvantages of Third Party Arrangements

Some of the possible disadvantages of using a third party could be:

• In-house staff morale will suffer, as they will feel they have to do the boring and less challenging maintenance work, but the exciting new developments and technologies are outsourced;
• The project Sponsor may abdicate her responsibility and accountability for delivery of the project, as it is assumed that the responsibility has shifted solely onto the supplier for delivery;
• The supplier may have over-stated its ability to deliver in order to get the business, resulting in under-performance against the agreed delivery dates;
• The customer may have inadequately specified the requirements to the supplier, resulting in increased scope and costs during project execution, which will be difficult to justify to top management;
• The supplier and customer may get into contract disagreements during the project, which could result in a souring of the working relationship and adversely affect supplier delivery;
• The supplier may have under-estimated the project effort and suffer financial loss, especially in the case of fixed-price contracts or contracts with penalty clauses. In extreme cases, this could cause supplier bankruptcy or else negotiation for extra funding;
• The customer does not control the supplier resources and these may be changed during the project to the detriment of the project delivery or quality.

This list of disadvantages is by no means exhaustive, but it gives an idea of some of the possible pitfalls that could befall a company embarking on a third party supplier project.

Importance of the Project Sponsor/Contract Manager

The customer company will typically appoint an internal staff member to manage the third party engagement. This person is critical to the success or failure of the project. They may take one of many attitudes towards the supplier, depending on their experience, skills and personal style. Some of the possible attitudes are:

• Abdication: the Sponsor expects the supplier to deliver against the agreed contract and abdicates all responsibility across to the supplier. The supplier must sink or swim and is not supported by the Sponsor during project execution, except by way of signing off agreed deliverables and approving progress payments in accordance with the contract. This may have the effect of alienating the supplier and causing an us/them attitude between supplier and customer staff.
• Adversarial: this attitude usually develops if there is perceived non-delivery from either the customer or supplier. An environment then prevails where the relationship is strained and driven through the specifics of the agreed contract, with neither party being prepared to compromise on any aspects. This does not make for a successful project and typically results in project delays and/or failure. In extreme cases, the supplier may decide to withdraw from the engagement and pay the required contract penalties rather than continue to completion.
• Partnership: the Sponsor recognises the need to incorporate the supplier into the company as much as possible for the project. Internal resources are encouraged to accept the supplier staff as peers and cooperation is encouraged and rewarded. This makes for a more productive working environment and reduces the risk of internal resources trying to sabotage the project. The focus is on the project team as a cohesive unit with common goals.

The Process

The process of a third party development project will typically follow these steps:

• Customer analyses and documents detailed requirements;
• Evaluate options and decide whether to develop in-house or to outsource;
• If outsource option chosen, develop a Request For Proposal (RFP);
• Develop criteria for suppliers;
• Identify suitable third party suppliers, based on the defined criteria;
• Send RFP to chosen suppliers;
• Suppliers present Proposals;
• Evaluate suppliers;
• Choose supplier;
• Negotiate Contract, based on the Proposal;
• All parties sign agreed Contract;
• Monitor supplier to ensure delivery against agreed Contract;
• Evaluate delivered outputs;
• Accept delivered outputs;
• Contract completion;
• Contract payment;
• Negotiate/agree Maintenance Contract, if applicable.

These steps could vary depending on the type of project, the complexity of the requirements etc. For example, where the requirements are not clear-cut, an initial contract could be entered into with the chosen supplier to determine the project objectives, scope and requirements. This could then be used as input into another contract for the execution of the project, based on the requirements determined in the first phase contract.

In order to highlight some common pitfalls which could result in project failure, the following sections will give some guidelines in contract establishment, supplier evaluation, supplier monitoring and deliverable acceptance.

Contract Establishment

Contracts for software development typically take the form of either fixed-price contracts or time-and-materials contracts. The fixed-price contract will bind the supplier to deliver the documented and agreed contract deliverables within the agreed timeframe and quality requirements for a certain total cost. All assumptions and constraints are specified, as these can affect the delivery by the supplier. The time-and-materials contract usually gives an estimate of the total costs for the project, but charges are made on the actual hours worked by the supplier staff on the project on a month by month basis. If the project runs for longer than estimated, the customer must pay the increased costs to the supplier.

The contract forms the basis for the project and needs to be detailed enough to ensure understanding of the project deliverables and deadlines, as well as the working relationship between the customer and the supplier. The contract should stipulate what project management reports will be provided to the customer on what regularity, the project scope and deliverables, schedule, cost, resources, roles and responsibilities, change management process, risk management process, assumptions and constraints. Many suppliers have “standard” contracts, but the customer company needs to analyse the contract carefully to ensure that it includes all the necessary detail for effective project monitoring.

A major problem that often occurs with third party projects is that the contract negotiation and changes are protracted, resulting in the project commencing before the contract is finalised and signed. This is a risky situation to be in for both the supplier and the customer, as there is not clarity on all aspects of the relationship and the project up front. This could cause disagreements, as contractual agreements cannot be enforced until the contract is signed.

Supplier Evaluation

The customer company needs to evaluate the capability of the potential suppliers to deliver the required project deliverables within the specified timeframe and quality constraints. This is not an easy task if the customer company has never dealt with the supplier previously. Possible aids to supplier evaluation are references from other customers, reference site visits and discussions, and formal supplier evaluations using frameworks such as the Capability Maturity Model (CMM) or ISO/IEC TR 15504.

Some suppliers may already have a CMM rating, which can be used as a guide to their capability to deliver on time and quality. If they do not have a rating, the customer company could request that an assessment be carried out to assess their capability maturity level. This assessment would typically have to be at the expense of the customer company. It could be done by an accredited assessor, or the customer company themselves could conduct an assessment based on an existing framework or their own framework. This would give the customer company a better idea of the supplier's potential to deliver against their proposal, since the assessment would highlight any shortcomings in the supplier's software development or related processes. Certain suppliers, however, may not agree to the customer company carrying out an assessment of their processes, as this may be seen as an infringement of confidentiality.

The writer has personal experience of where a supplier assessment has been used to mutual benefit of the customer and supplier companies. An existing supplier granted permission for the customer to conduct an assessment of their development processes, to highlight shortfalls and strengths and to recommend improvements. The resultant analysis was then used to implement the improvements, which were ratified through a follow-up assessment a year later. The subsequent project that was carried out utilising the improvements showed improved delivery and processes. This has resulted in a strengthening of the partnership relationship between the customer and the supplier.

Supplier Monitoring

“The purpose of the Supplier Monitoring Process is to track and assess the performance of the supplier against the agreed requirements” (ISO/IEC TR 15504-2). The process therefore needs to have agreed standards, which are detailed enough to track performance, as well as agreed timelines and quality criteria for deliverables. Whether the contract is fixed-cost or time-and-materials, the customer needs to monitor the supplier progress to ensure that the project delivery and costs are not at risk.

The desired outcomes are that, as a result of the successful implementation of supplier monitoring:

• joint activities between the customer and the supplier are performed as needed;
• information on progress is exchanged regularly with the supplier;
• performance of the supplier is monitored against the agreed requirements; and
• contract changes, if needed, are negotiated between the customer and the supplier and documented in the contract.

In order to achieve these desired outcomes of the process, it is essential that certain actions are taken and inputs and outputs produced to achieve successful completion of the process. These processes are discussed briefly in the sections below.

Contract Management

The contract is the basis for the engagement and needs to have the required detail to enable effective management of the project. Any changes to the contract need to be controlled and signed off by all the relevant parties before becoming part of the contract, whether in the main body or as an Addendum. The initial evaluation and agreement of the contract and any subsequent contract changes need to be done with legal consultation, as the cost and other impacts of contract breaches need to be understood and agreed. The contractual aspects of the outsource arrangement are so important and integral to the process that some customer companies have a specialised function to manage project contracts rather than having this as one of the responsibilities of the project sponsor or project manager.

Requirements Specification

The Requirement Specification will either form part of the contract or be done as a separate approved document, such as a Statement of Work. This forms the basis for the project scope, as agreed between the customer and the supplier, and as stated in the deliverables list. Each requirement must be uniquely identified and verifiable as complete and correct, once completed. Any changes to the requirements need to be documented and approved by both parties before being included as part of the project. Typically the cost and schedule impact of such changes will be determined before any decision is made for their inclusion or exclusion from the project scope. A method of sizing requirements that can be very helpful is Function Point Analysis (FPA). Some suppliers use this method to size and cost the development effort, as well as to size and cost extra requirements that surface during project execution. A different cost per function point is normally applied by the supplier, based on the project stage in which the extra requirements are requested, as the cost to make changes increases as the project progresses towards completion.

Work Product Standards

From the outset of the project, work product or deliverable standards need to be documented and agreed between both parties. This is to prevent any misunderstandings or disagreements as to what is meant by any of the agreed deliverables that form part of the project scope. The standards will include the quality criteria and format for each deliverable, as well as the reviewers and approvers. If any deliverables do not conform to these agreed standards, then they will be rejected by the customer and the supplier will have to rectify the offending deliverables before acceptance will be given. In the absence of specific work product standards, the customer will have no grounds for rejecting any of the supplier deliverables.

Progress Status Report

The parties need to agree on the format, content and frequency of the Progress Status Report. This is the main source of evaluating project progress against the baseline target dates for delivery, and as such is a vital communication and control document. If the progress reporting is vague and incomplete then the Sponsor and other key project stakeholders will have no way of objectively assessing project progress and of taking pro-active action to prevent project slippage or cost overrun. As a minimum, the Progress Status Report should include the baseline start and end dates for major deliverables and milestones, baseline costs, actual costs to date, forecast and actual start and end dates, approved changes, risks and their mitigation actions, and issues. It could also include a summary project health assessment for schedule, cost and quality to give an overall view of the project health.

Meeting Minutes

Regular, well run meetings with the supplier are essential for communication and on-going project co-ordination. During meetings many decisions are made and courses of action discussed and agreed. If these are not documented as formal meeting minutes, then disagreements can arise afterwards as to what was agreed. To safeguard both parties, as well as make for more effective project execution, minutes of all meetings and workshops must be documented and agreed by all parties. These form the basis and a history of project decisions. The minutes should specify who was present, the decisions made and all discussion points. From the meeting minutes, other documents may need to be created or updated accordingly, such as scope change requests, issues log, risk log, and the contract. The discipline of formally documenting and agreeing meeting minutes is one of the most important aspects of supplier management, but its importance is often under-estimated, except when a dispute arises.

Change Request

Any changes to the agreed parameters of the project, such as scope, requirements, schedule, cost or quality, need to be documented on a formal Change Request and approved by all key stakeholders before they form part of the project. The impact of any change needs to be assessed by the relevant project team members before it is presented to the Project Steering Committee for approval/rejection so that all parties are aware of the implications of the change on the project. If changes are not controlled in a disciplined way then the probability of the project achieving its stated objectives is greatly reduced. If a change is approved, then the project plan baseline can be updated with the agreed change impact, to enable monitoring against the new baseline from that point forward.

Communication Mechanism/Plan

All the project stakeholders need to be kept informed of the project issues, progress, and all relevant project information to enable them to act on the issues that pertain to their area of responsibility. The Communication Plan must outline what information will be communicated to each of the stakeholders during the project to enable them to perform their respective project roles effectively. The customer must ensure that the supplier has identified all project stakeholders and adequately planned for an effective communication mechanism to achieve this. The plan must include the media to be used as well as the frequency of the communication. For example, the project progress report may be specified as an electronic format report to be distributed to all project stakeholders on a monthly basis before the last working day of every month.

Problem Report

All identified errors with the deliverables and functionality of the system need to be logged and managed to resolution. A mechanism to facilitate this process needs to be agreed with the supplier and utilised throughout the project. If this is not done, then control of the issues and outstanding work will be impossible, thus leading to misunderstandings and project delays. The problem report system will be used by the supplier to report back to the customer on the status of problems and their expected resolution dates. This also forms the basis for the decision as to whether the system is ready for implementation or not, as the presence of many unresolved errors could delay the implementation date.

Review/Acceptance Record

The crux of performance against the agreed contract is the delivery of the agreed outputs that satisfy the agreed quality and acceptance criteria of the customer. A record must therefore be kept to keep track of which deliverables have been approved or rejected and the criteria for acceptance. This ensures no misunderstandings regarding the achievement of the objectives against the contract. If a deliverable has not been accepted by way of a formal sign-off by the designated responsible person, then the task to produce that deliverable is deemed to be incomplete. Progress against the plan is therefore measured by signed off deliverables and not merely by the tasks completed, as this could present a misleading picture of progress.

Deliverable Acceptance

The purpose of the Deliverable Acceptance Process is to approve the supplier's deliverables when all acceptance conditions are satisfied (ISO/IEC TR 15504-2).

As a result of successful implementation of Deliverable Acceptance:

• acceptance will be based on the acquisition strategy and conducted according to the agreed acceptance criteria;
• the delivered product and/or service will be evaluated with regard to the agreed requirements.

In order to achieve this, the customer needs to evaluate the delivered product to see whether it satisfies the agreed criteria, as specified in the contract, and then to sign off the deliverable as completed in accordance with the criteria.

Acceptance Test Strategy

This specifies how the customer will validate that the delivered system satisfies the requirements, as stated in the Requirements Specification. The acceptance criteria will include criteria for each deliverable, as well as the test plan and expected results for the functionality of the system. The customer must ensure that these test plans include sufficient test coverage to verify and validate the system against the requirements.

The test strategy should include non-functional testing, criteria and sign-off. These will be aspects such as agreed performance criteria and system usability criteria. Otherwise the system may fulfil the functional requirements specified, but be rejected by the end users due to inadequate performance or poor user-friendliness.

Sample Case Study

The writer has been involved with a third party supplier to his company for the past 5 years. The supplier was contracted to supply a Financial Needs Analysis system for the customer's base of insurance intermediaries. The initial project ran for double the estimated timeframe and the resultant implementation was deemed to be unsuccessful because many of the potential users did not take up the software. Those that did use the software were not happy with it and many complaints were received, due to the unreliability and certain errors in the software. The writer got involved in a subsequent project to rectify this situation and to attempt to realise the originally proposed cost/benefit of the software. The project was managed in accordance with the framework above. Due to the failure of the previous project, the relationship between the customer and supplier staff was not good. The writer took this into account in the strategy for the project and as part of the stakeholder management.

One of the aims of the project was to re-establish a productive and co-operative working relationship between the supplier and the customer, so as to move from an adversarial relationship to a partnership relationship. The project kicked off with a combined workshop to agree on the scope of the project and gain a common understanding of the requirements. This was well received by all parties. The contract was then drawn up, based on these agreed requirements. The supplier agreed to fix the cost to the customer and to carry some of the costs themselves, as they also stood to gain from the project as they could provide some of the functionality to their other customers.

Project management standards and procedures were laid down and agreed. This resulted in far improved communication and control of the process. Issues and problems were logged and managed effectively. Progress was reported regularly and concisely, which was not done in the previous project. All meetings were minuted and communicated to all stakeholders, resulting in a common understanding of the issues and pro-active resolution of issues and mitigation of potential risks. The users were also better engaged in the process and the training process was improved over the previous project, as this had been identified as one of the major causes of user dissatisfaction and resistance.

The resultant implementation was well received by the users and a turnaround in user attitude was observed. The relationship with the supplier has been greatly strengthened through this process and is now seen as a partnership with common shared objectives. The customer has also performed capability maturity assessments on the supplier processes over the 5 year period and seen a marked improvement in the ratings as a result of the implementation of the improvement recommendations by the supplier. This shows that the application of these principles to third party supplier management can lead to better results in the long term for both the supplier and customer companies.

Conclusion

This paper gives guidelines for companies engaging third party suppliers to do software development for them. The management of third party suppliers requires the application of certain processes and principles in order to reduce the risk of project overrun or failure, or the risk of irreparable harm to the customer/supplier relationship. By applying these guidelines the customer has a marked increase in the probability of project success and a mutually co-operative customer/supplier project environment.