judging the potential for excellence
by Anthony P. Venturato
As we move toward a greater reliance on design/build contracting, simply perusing a proposal is insufficient to correctly judge the prospective quality of work and the ability to perform, which lays the foundation for a project's success. When choosing firms and individuals to build major public projects, qualification concerns are more relevant than ever.
Traditionally, hiring consultants to plan, design or manage construction of public works has been based on subjective evaluation criteria. Construction contractors, on the other hand, are normally selected based on lowest bid. Lately, however, the trend toward design/build projects (turnkey), negotiated procurements, partnering, guaranteed pricing through early owner/contractor relationships, and other innovative project management techniques have obscured the distinction between selections based on qualifications and those based strictly on price.
Nowadays price is often simply tacked on to the list of criteria to be considered to evaluate a company's qualifications. Price is, by its nature, a readily quantifiable and comparative criterion, whereas evaluation of qualifications is not. What follows are suggestions for evaluating public works proposals on the basis of qualifications; many apply to private ventures as well.
Qualitative vs. Quantitative. Let's address the system of evaluations and ranking first. Usually, a selection panel, consisting of several technical and commercial people appointed by the owner, reviews the proposals, compares findings and consolidates its evaluations into a recommendation for award. Before the evaluation and, in fact, outlined in the Request for Proposal (RFP), a ranking of relative importance, determined by the owner, is assigned to pricing factors and qualifications (as will be shown below, selection of the best group of individuals normally should be the owner's top priority).
The Qualitative Evaluation Process: Key Steps
- ∎ Establish evaluation criteria in order of importance to the owner. Personnel with relevant experience should be at the top of the list.
- ∎ Appoint selection panel who are technically and commercially equipped to represent the owner's best interests and are predisposed to working together as a team.
- ∎ Detail and document all proceedings that follow. Keep the records of the panelists’ assessments and ranking as the process develops.
- ∎ Each panelist reviews proposals and ranks them from best to worst. [Do not use a point system.]
- ∎ Selection panel meets to compare findings, deliberate issues that arise, and adjust rankings based on the discussions.
- ∎ Selection panel determines strategy for researching the proposers’ claims and personnel backgrounds. Panelists place calls to previous proposer clients.
- ∎ Selection panel meets to discuss findings, adjust rankings, and develop questions and issues to pose to each proposer during the interviews.
- ∎ Selection panel conducts the interviews.
- ∎ Selection panel reconvenes to finalize rankings and determine the apparent winner. [The quantitative element (price) and the qualitative elements are considered in combination in design/build contracts.]
- ∎ Owner negotiates contract, ties up any loose ends, and ensures commitments. [If negotiations are unsatisfactory, commence negotiations with the runner-up proposer.]
- ∎ Execute contract upon successful negotiation.
Because evaluation of qualifications is highly subjective and difficult to quantify, assigning a numbered criteria and scoring system to this kind of competition and evaluation process will likely introduce errors in conclusions about relative merit. Accordingly, the outline of evaluation criteria in the RFP should be stated in relative, rather than numerical, terms. For example, the owner could state in the RFP that the most important criterion is experience of the proposed individuals, followed by construction pricing, followed by applicable previous projects performed by the proposer companies, and so on . ^ The eventual ranking among the various proposers based on the criteria should also be kept to a relative rather than numerical process.
The surest way to determine the most qualified proposer is to rank them directly against each other, i.e., keep the comparisons qualitative. This also aids the panel discussion by compelling each panelist to speak to the reasons why he or she feels one proposer is better than another rather than trying to justify a particular point score designation. Evaluations are much more effective when one can argue that Proposer A is better than Proposer B because of this or that experience rather than arguing that Proposer A was given 8 points and Proposer B was given 6 points. Point systems become a numbers game, mislead and bog down the evaluation exercise because the numbers don't accurately represent the situation. (In one actual case, a selection panelist gave his favorite proposer 100 points and all the others 0 to skew the overall scoring his way. In another case, a mediocre proposal won out simply because its score in all categories was fairly even and in the aggregate happened to tally higher than the better proposals, which were scored unevenly by the panelists, reflecting their divergent opinions.)
Conversely, relative ranking kept in qualitative terms can be adjusted by the panel as a group exercise based on dialogue. The selection panel proceedings should be fully documented in order to provide the owner with a clear explanation of the final ranking.
Beware of Pitfalls! Now that we've covered process, let's get to the main theme: How does one determine the best qualified for the job? The answer seems simple: The proposer that has the most relevant, successful experience. Searching for relevant experience in the proposal may tell you the proposer's past jobs but it will be difficult to tell how important the particular role of the proposer was. For example, a proposer's claim to having designed the transportation system of a major project may be, upon closer examination, nothing more than a minor traffic study and development of recommendations that may or may not have been implemented. Remember always that a proposal is a self-serving document; every statement in it is cast in a favorable light.
Another aspect that could mislead is the proposer's understanding of the project being proposed upon. This will only give the reviewer a sense of the proposer's ability to gather information about the job and feed it back to the owner. Let's face it, understanding of project requirements will come soon after contract award to any proposer. Unfortunately too many owners put a lot of stock in “Understanding the Project Requirements.” This pseudo criterion will only tend to work against fresh talent. Instead, such a criterion should either be eliminated or changed to read: “Demonstrate that your previous experience relates to this project.”
Perhaps the most serious failing of all is insufficient research. Too many owners rely solely on the proposals without contacting the proposer's previous clients. I recall one owner who stated with conviction that he had chosen the best consultant in the world. When asked why he thought so, he replied with a straight face that the consultant had told him so.
How to Determine Relative Experience. Investigate the individuals proposed. The experience of the proposer firm takes a back seat to the key individuals proposed. The fact that a firm designed the world's longest bridge or the tallest building or the deepest tunnel is impressive, but what have they done lately and are the engineers involved in those landmarks being proposed on your job? The owner should scrutinize the project managers, lead engineers, construction supervisors, and key support personnel proposed. When you get right down to it, the firm doesn't design and build the job; the people do.
How did they perform when the going got tough? This gets to the heart of the dependability issue. All projects experience problems; the owner will need people who are good at getting projects unstuck. Look for individuals who are not only good at avoiding problems through good planning (basically from having done it before), but who are also good under pressure. The owner should seek out experts who have overcome such challenges in order to maintain schedule and budget.
Checking on reliability, maintainability, and operability of the installed systems is also a good idea. It's important to verify the candidates can design and build well, but it's also important that the finished product they have designed and built works well and meets or exceeds expectations over the long haul.
Old Flames. The most reliable way to determine the answers to these questions is to seek out and talk with those previous clients—in the plural (and don't trust just the list of references submitted with proposals). Ascertain who the previous owners’ representatives were or, better yet, talk with the owners themselves. Ask them about dependability of the individuals assigned, their flexibility, creativity, professionalism. See if the communication links were good.
Along the way, find out something about the firm's dependability. It is important to know if the firm is going to back you in tough times, too. There have been cases when consultants not only were slow to come to the rescue but even demanded change orders be agreed upon before they would lift a finger.
In the Owner's Interest. Some attention should also be paid to organizational approach and backup resources. An overly complicated organizational structure could be cloaking a billings refuge for otherwise idle workers.
Backup resources are important to those owners who foresee the job as very complex and prone to outside or unpredictable influences. In those cases there would be a certain value attached to a company or team with a large pool of professionals available in the home office, even though those experts are not being proposed as directly assigned to the tasks outlined in the solicitation. However, even a minor deficiency here can be remedied as part of the proposal process. For example, the proposer may be persuaded by the owner to line up other subcontractors in advance or at least identify such a resource based on previous working relationships with other individuals or companies.
Confronting the Issues Head On. Remember, during the oral interviews, you aren't screening for a role in the movies. Glittery presentations and special effects are fine entertainment, but style is a fleeting attraction. Substance is what's important. The selection panel, having reviewed the proposals in-depth by now and done extensive research, as suggested above, should collaborate among themselves just prior to the orals. The purpose of this get-together is to develop hard-hitting questions to pose to each proposer during the oral interview to cover any loose ends or issues that may have arisen during the research or just to gain clarifications. Keep in mind, however, this isn't a gotcha quiz; the selection panel should endeavor to obtain all information necessary to make the right decision. Subsequent to the orals, the selection panel should again convene to determine whether rankings should change as a result of clarifications or new information derived from the oral interviews.
Tally the Scores. You need to link the qualitative findings to the quantitative element, price. Once you have a ranking of qualifications as well as a sense of the margins between the competitors, e.g., the first ranked is much better than the second or the first and second are a close call, etc., hold the list up to the bids. If the best qualified is also the lowest priced, the competition is over; sign the contract. Unfortunately, Christmas only comes once a year; usually, the owner has to strike a balance between qualifications and price. In dealing with this difficult situation the selection panel will be tempted to revert to a scoring system, such as say 55 percent for quals and 45 percent for price, or some other formula that will conveniently solve the problem. Resist the temptation if you can. Although it makes for more heated debate if you maintain the basis of the arguments in qualitative terms, the deliberations are well worth it.
Keeping in mind that conditions and requirements will change over the life of the project, and that problems or even crises will arise from time to time and have a great effect on price if not handled properly, the qualifications of the proposers should carry more weight than bid price. However, this is a judgment call based on the particular needs of the owner, and their relative importance should be clearly stated in the RFP.
No One Said It Would Be Easy. It gets more complicated and needs more discussion among selection panelists when the qualification ranking and the bids are very divergent. The debate may get intense, but such value judgments deserve attention. Moreover, negotiation strategies will likely result from the selection panel deliberations because these situations usually necessitate one or two more rounds of talks with proposal finalists to bring rankings into better focus. This in turn can generate a benefit in the form of reduced prices or further improved proposals.
The final step in the process is negotiations. Don't dismiss the second or third ranked proposers yet. Let them know that a tentative selection has been made to enter into negotiations and that they would be informed of the final outcome afterwards. Also advise that each in turn may be called upon if negotiations with the top-ranked proposer break down. This provides a fallback and places the top-ranked proposer on notice that the owner expects good faith negotiations.
The owner's panel should keep in mind throughout the selection process that the choices being made can mean success or failure of the project, because it is much too difficult and costly to fix later. The design and construction industry is moving toward consolidation of work, as shown by the present shift toward turnkey contracting (design/ build). The owner should be very fussy about who gets to do this work, particularly since such contracting means the owner will be relinquishing a great portion of the control to the contractor. If there are any weak players (firms or individuals) in the top-ranked proposer's team, the owner forthrightly should require changes prior to signing on the dotted line.
At the risk of stating the obvious: Beware the old “bait and switch” tactic. Strong prohibitions should be incorporated into the contract to prevent the contractor from substitutions, particularly of agreed-upon personnel, without approval of the owner. It should be made very clear to the proposer that the only way to get out of a personnel commitment is head first or feet first—thrown out or carried out.
Documentation is Important. Maintain fairness and give all points of view a proper hearing throughout the evaluation process. Document all proceedings thoroughly. Prepare a comprehensive report, including dissenting opinions, detailing how the selection panel arrived at their conclusions every step of the way. This is particularly true for public agencies, where protests from proposers who weren't selected often occur.
GOOD LUCK! Some bad players may slip through the net no matter how thorough your effort, but following these commonsense principles should make it less likely. ∎
Anthony P. Venturato is a project manager for STV Incorporated, Redmond, Wash. He has a background in project management and executive administration of complex, large-scale programs in mass transportation and infrastructure projects.
This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI.
PM Network • March 1997
PMI research shows project teams that draw from an array of perspectives and skillsets deliver powerful outcomes.