The single greatest decision on many projects is selecting a vendor to provide software or services. The methods used for soliciting, evaluating and negotiating with vendors, however, are often not reviewed later for their impact on the eventual success of the project. This is especially the case when the decision process has not been done “according to the book,” which is when the results of the decisions made are most likely to result in problems during project execution.
This paper demonstrates a case study contrasting two similar projects to propose a method to take stock of the actual solicitation and source selection processes while a project is still in process. This will help a project manager find warning signals early, allowing them to identify, and manage significant project risks that otherwise might be overlooked or minimized.
This topic is especially meaningful for project managers who are assigned to projects that are already under way—which seems to be all of us, at one point or another—or who are in an environment in which decision-making is performed in an ad-hoc fashion. This topic is also relevant to vendors by underscoring how certain actions they take will throw marketing promises, customer expectations and project execution out of alignment, leading to dissatisfied onetime customers at best and lawsuits at worst.
Background
My interest in this topic is highly personal, having been placed between a rock and a hard place once too often. My case study focuses on and compares two application development efforts that I was hired to manage simultaneously, after all the major decisions had been made. I learned much that was highly revealing about the decision process. Even at that point, the lessons learned in these interviews clarified the projects' current status, and I could make predictions, which were all too accurate, about project results.
Procurement by the Books
The Project Management Institute's A Guide to the Project Management Body of Knowledge (PMBOK® Guide) lists the four initial (pre-execution) processes of procurement as: Procurement Planning, Solicitation Planning, Solicitation, and Source Selection.
The inputs to Procurement Planning include the project scope statement and information on available resources and market conditions, and are used to drive a build-or-buy analysis, which results in a procurement management plan and a Statement of Work (SOW). The SOW is used as input to the Solicitation Planning process, during which a Request for Bid or RFP-like document is developed to elicit vendor proposals, in addition to the criteria to be used to evaluate vendor responses or software.
During Solicitation, the goal is to locate and receive proposals from appropriate vendors. Within the Source Selection process, these proposals or software alternatives are assessed against the evaluation criteria, and contracts are negotiated with the vendor(s) selected.
Procurement in the Vernacular
In organizations with a Procurement organization or a Project Management Office with experience in procurement support, these steps or something like them are taken on all projects above some budget limit.
On the other hand, in many organizations, and especially on projects given special leeway because of a perceived pressure to act quickly due to market demands, teams are allowed to follow their own process in what is often a pro forma but inadequate evaluation.
The minimum set of steps for any evaluation is:
(1) Define requirements / initial scope, (2) Research buy vs. build, (3) Prepare summary of requirements / RFP / list of selection criteria, (4) Determine vendors, (5) Evaluate options, and (6) Negotiate contract.
If shortcuts are taken at any one of these points, and especially if they are taken at several points, the result of such an evaluation will be a project wide open to high risk situations, as we shall see.
Case Study—The Situation
My study focuses on two information technology projects, which I was hired to manage simultaneously. These projects were part of an ambitious program to replace existing software applications for corporate clients of a major services company with web-based applications. A primary motivator for the business units involved was to keep up or catch up with the primary competitor in their market, so in the name of “time to market,” two vendors were engaged to modify or build upon their own software.
Exhibit 1
I was brought in after the requirements definition, after the vendors were selected, and after the solution was determined, in order to oversee the implementation of these two systems. It was quickly apparent that these projects were heading in two different directions. Whereas on one project there was smooth sailing, on the other, the vendor had just survived the suggestion by the chief technologist on the project that they should be removed. His response was to disappear into other projects on which he was engaged.
Needless to say, I more than had to hit the ground running. After some time, I arranged to interview the team members most engaged in the vendor selection process, using the truthful premise of an assignment for a project management course in Negotiations and Contracts, which I was taking.
It was my hypothesis that looking into how these vendors were selected would shed insight on how each project had gotten to where it was at that point. Some of what I learned—including details that had not been conveyed to me previously—were highly predictive of the end game of these particular projects. One of my lessons learned, and an intended take-home from this paper, is that such an investigation should be part of the immediate orientation process of any project manager coming onto a project already under way.
We will look first at what my interviews told me about the selection process, and then at the vendors' negotiation stances. Lastly we will look at “how things were” at the time of the interviews.
By not initially identifying which components go with which, I hope to encourage an exploratory point of view. From standing back for the few moments it takes to read the case studies, it should become clear that focusing on the wrong selection factors—just as the client do—can have disastrous results.
The Interview Process
In preparing my case study, I interviewed the primary participants on each project, including:
1. Products Program Director—non-technical proponent and program manager over multiproject web initiatives slate. Set up aggressive dates prior to sign-off on requirements or initial planning.
2. Technology Group Manager—also non-technical, reports to CIO. With the Products Program Director is point of contact between projects and Project Review Board.
3. Manager of Web Development and Infrastructure—technical but advisory role (close personal friend of CIO). Disappeared from troubled project when disagreed with vendor selected.
4. Primary Business User. Each primary contact was only parttime on their respective project.
Depending on preference of each interviewee, I used some combination of email inquiry and response, phone conversation and inperson interview. I used a small number of specific but open-ended questions and followed up based on their responses.
My primary three questions were:
1. Why was this vendor/solution selected?
2. What factors weighed in their favor?
3. What factors weighed against them?
The first question explored why, from each participant's point of view, the vendor was selected, and what factors were considered most significant in making this decision. Follow-up questions attempted to flush out what alternatives had been looked at and how.
The second and third questions inquired about what factors were accentuated, and which were downplayed or ignored. When the participants indicated that there were factors that weighed against a potential vendor, I asked them to clarify what the vendor did or said, which caused them to overlook or downplay the perception of a possible problem.
Exhibit 2
Finally, I also asked the participants about their understanding about how other members of the team had made their decisions, and verified what I understood about how the vendor had participated in the process (such as who had participated and what kind of documentation or proposal had been required).
What I heard correlated closely to problems, limitations and strengths that were already in evidence in the execution phase of these projects. There were critical factors in the approaches taken by both the client and the potential vendors, which were ultimately accurate indicators of how the projects would continue.
Interview for Case A: The Vendor With Only Plans for a Product
In this case, a product was needed to create the web-based front-end and a set of reports over a back-end product then being implemented (in a major conversion from multiple existing applications). The vendor selected was the company providing the backend product, in spite of the fact that the product was still under development and the initial feature set fell quite a bit short of the specified client requirements.
The matrix in Exhibit 1 provides the answers I received with respect to this evaluation and selection process.
“Time to market” was the predictable mantra driving this decision, as the front-end needed to be in place to match the anticipated roll-out of the back-end, but the key factor seemed to be the hope of a seamless approach, and minimal demands on the client team members during development. No other alternatives were considered beside “build” and that was rejected due to lack of expertise with the new backend.
Interviews for Case B: The Vendor With Installed Products
In this case, the vendor had provided (and still continued to support some three years later) a part of the complicated back-end system for which the web-based interface was to be developed. Because there were several components of the system, and because eventually it would interface with an overseas system as well, an intermediate database was to be designed by the client staff, against which reports for the web would be run by the new software.
In addition to having previously supplied the back-end product at this client, the vendor indicated that they had installed a similar application at this client's major competitor. Because of this, in the “buy vs. build” discussions for this project, the proposal from this vendor was treated by the non-technical managers as a “buy” with minor adaptations, because it was assumed that the preexisting report generator was the essential feature of the application being purchased. The chief technologist disagreed with this viewpoint.
Not surprisingly, “time to market” was the story here as well. In this case, the big plus was the expectation that there was a database-oriented report-writing tool already available, and the big minus was that previous work with this vendor had performed poorly in the deployment of its previous product.
Vendor Negotiation Stance
Before we look at which of these two projects was in trouble and why, let me add just a bit more information here. During these interviews, and based on what I saw in their interactions with my teams and myself as we (belatedly) prepared project plans, determined milestones and dove into detailed design, I learned a few things about how each of these vendors had approached and acted during the shortcut negotiation phase of each project.
Vendor X
• Negotiation was handled as sales maneuver, and salesman (a company principal) made promises based on his own inaccurate technical understanding, which confused users and other team members alike.
• Focused on influencing client's perceptions of vendor's ability to do work, and on minimizing change required and time it would take. Vague definitions of promises, time, and money.
• Demonstrated technical hubris: vendor apparently did not own (or know?) their limits with respect to web experience.
• Client-server software pushed to deliver reports of the size needed. This would have to be installed on each desktop of the client's institutional clients. Though disallowed from original proposal, was re-proposed every time the vendor hit a snag due to their lack of expertise in web environment.
Vendor Y
• Negotiation started from vendor's own strategic decision about growing product in joint venture.
• Identified customer needs with respect to product; demonstrated competency and willingness to bend.
• Stuck to own limits—offered what could do and no more; clarity about internal resources availability.
• Everything requiring additional money spelled out precisely; some “freebies” included but all decisions based on technical and time requirements, nothing meant to hook client.
• Proposed defined process for requesting, negotiating, and signing off on enhancements.
Last Clues: Where Did the Projects Stand?
For fairness, I am situating you where I was at the time of this investigation.
Project 1
• Project proceeding as “black box” development. Vendor/client meetings focus on inputs, outputs and display. Client not pulled into details of design or development process except where necessary.
• Meeting high percent of deliverable dates—following through with requested changes.
• Change of infrastructure location without notice and additional constraints resulting from this had significant impact; vendor offered resources to help work through issues.
• Solid processes; project lead at vendor from technical side originally but was proficient as project lead for own staff, provided liaison for all discussions.
• Other clients had purchased product meanwhile.
Project 2
• Project plan as provided included 20 lines, missing many deliverables and entire phases. Project lead (no prior management experience) spent 95% of time on development. Estimates were 100% inaccurate.
• Poorly designed software-generated web front-end had never been seen by clients; time for designing an appropriate interface not in original plan and vendor tried to argue not in scope and unnecessary.
• Client had to tap resources to attempt to fix design errors. Project manager had to mentor vendor on meaning and process of “system testing” and “integration testing.”
• Vendor attempted to train personnel as they went rather than bringing in appropriate expertise.
• Deadlines inevitably missed liaison was usually sales representative who was unable to get good information from his people and made promises without understanding technology.
• “Buy” option turning into full equivalent to “build” time frame, including high demands on time of business users on the team (which had to be expanded to account for these needs).
By now, you can probably see that Vendor X (negotiation as sales event) matches with Case B (poor reputation), with results in Project 2 (time overruns and high demands on client team). They had sold themselves as providing a customization of an existing report-writing product, due to the assertion that their product was in use at a competitor—which turned out to be false, as the users there refused to use it. The client had overlooked their own bad experience with the development, testing and deployment practices of the vendor, based on their assertion that they would do it better this time.
Vendor Y (negotiation as strategic partnership), on the other hand, with Case A (no software to begin with and no clients), had the results shown for Project 1: they bit off only what they could chew, had sufficient processes and staff that utilized them—and ended up providing working software that, meanwhile, was being successfully sold to other customers owning their back-end product.
Vendor X was removed from the project eventually, due to constant schedule slippage and their inability to produce software that the client was willing to deliver to their own institutional clients.
Lessons Learned
What I discovered was that a nominally appropriate selection process had been utilized:
1. The software selection team combined technical, business users and management.
2. High-level requirements were defined in advance (though no separate selection criteria).
3. Potential vendors were identified and a short-list was made based on requirements.
4. Team reviewed offerings from short-list vendors and made selection.
5. Contracts were negotiated.
This looks well and fine. Let's look at some of the problems that appeared upon closer look:
1. The selection was not lead by a project manager, nor used effective decision-making techniques.
2. The requirements had not been signed off prior to selection.
3. The “gap analysis” used to compare requirements to vendor offerings used vague definitions.
4. There was only one product pursued for each project. The question asked was, can they do this better than we can? But not, can they do the job in time they say?
5. Vendors not required to prepare detailed proposals or substantiate their claims. Custom report design was OK'd on the gap list by the stated assumption that “the flexibility of the vendor's reporting tool will handle this”—without proof.
6. Due diligence on vendor not performed, references not verified, and past experiences minimized.
7. Contract negotiation was not completed prior to starting projects. No more attention was placed on style of negotiation by vendors than on the substance (or lack thereof) of their offerings.
8. Since managers who served as liaison between team and review board had pushed selected product, they minimized problems to upper management, and insisted to team the decision was irreversible. The final decision to replace vendor took a mini-revolt by users and technologists.
What Happened Here?
• The client was eager to get an overly-ambitious catch-up effort going, committing too few internal resources across too many projects. The emphasis on time-to-market overruled common sense.
• “Buying” a product sounds faster than “building,” but with both projects, an incomplete product required time for design and development, and the original project plans reflected that fact in only one.
• No project manager meant key aspects of procurement planning and management were ignored.
• My investigation came too late. As much as I was thrown into the fire immediately, a more thorough investigation as soon as I arrived would have allowed me to identify and plan against more of the gaps—i.e., not knowing that the front-end needed designing meant deliverables were missing from my plan.
Investigation Process for Newly Arrived Project Managers
1. Diagnosis: Use interview or other discovery process to find out how the decision was made, who was involved, what processes were or were not used. Were meetings used, predefined criteria, consensus? Learn to recognize warning signs that predict potential problems.
2. Look out for “weak points” in team or vendor attitude: Anything defined as a stop-gap measure “until we have time to do it right”—or any particular phrase used over and over that indicates common sense may have been overruled—these will lead you to areas where pertinent information may have been overlooked or ignored.
3. Become alert to Critical Success Factors. If these were not defined or were left vague, you need to get concrete answers before you proceed.
4. Perform risk identification and analysis against any factors found in first three steps. Include the team, and make sure everyone knows these are risks. Make the risks measurable in some way, even if full quantification is seen as too time-consuming or irrelevant “because decision made.”
5. Highlight the potential showstoppers and their warning signs. Predefine tolerances to trigger closer attention; get sign-off if possible but in any case include in reporting.
6. Strategize on ways to track and respond. Include vendor and team if possible.
7. Define “warning light” and “go/no-go” triggers: Develop with team, keep handy and use. Include variances against Critical Success Factors, not just progress against deliverables!
Recommendations
• Call a “time out” as soon as you arrive to review, replan, and fully acquaint yourself with the reality of the project you are joining. Chances are there will be plenty of time for heroics later!
• “Coming up to speed” on a project should include all documentation. In addition to business case, requirements, scope documents, look at build vs. buy, gap analysis, even meeting notes for hot spots.
• Be willing to probe and re-lead thinking on key points. The importance of “time to market” (or lowered costs or whatever the current mantra is) probably is not important enough to alienate ones market by releasing a poor quality product!
• If at all possible—be there to lead evaluations and use “effective decision-making techniques”! If you weren't there, incorporate the techniques into review and clarification meetings/investigations.
Effective Decision-Making—The Alternative
• Well-defined problem
• Predefined selection criteria
• Decisions include right people, and are made in own time and place
• Communication is emphasized
• Decision-making tools and techniques used to focus on evaluation and selection
• Results used as feedback to the process (CSF and decisions tracked to completion)
• Structured decision-making steps used:
1. Identify problem.
2. Collect information.
3. Identify feasible alternatives.
4. Evaluate alternatives.
5. Select the “best” alternative (and be able to support decision).
6. Implement and assess results.
Recommendations to Vendors
• Be wary of who represents you and how; don't let someone sell you as a cookie company if what you make best is ice cream!
• If you decide to go into the cookie business—hire some bakers.
• Cookies or ice cream—don't offer to sell more than you can make. Better to under-promise and over-perform than to over-promise and under-perform.
• Demonstrate your project management processes rather than promising them.
• Know your client's Critical Success Factors—including the unspoken ones—and be able to meet them.
Take-Homes
A project manager—even coming in midstream of a project—can use a proactive discovery process, such as I have described here, in order to identify and analyze a wide range of risks associated with the vendor choice and create appropriate mitigation strategies.
First of all, what you don't know can definitely hurt you, and secondly, though you cannot change what is done, you can increase your awareness of risks. And finally, you can use your own discovery process as a way to bring a greater level of awareness across your team to the possible consequences of choices made.
For project managers in stable project-oriented environments, the two take homes may be to remember to include good decision-making processes on their projects, and to ensure that the processes for Solicitation and Evaluation are monitored on difficult projects, with an eye to spotting weaknesses mentioned here.
Hopefully those who take the most home will be those faced with a “project from hell,” as this is where these techniques will be most needed. Equip yourself with a better understanding of your project's early phases, and start a pattern of communication that does not shy from dealing with how things are, and you should be better prepared to lead your project to success, in spite of whatever has come before.
