Opportunity assessment – before it is a project
Michael O‘Brochta, Senior Project Manager, Central Intelligence Agency
The future of successful project management involves doing the right projects—not just doing projects right. How can we tell if we are working on the right project? This paper presents a best practice approach for assessing an opportunity before it is a project and shows how Opportunity Assessment can be used to improve the odds that we are in fact working on the right project and that it will succeed. Industry surveys and statistics are used to describe the primary reasons projects do not succeed and to make the point that the earliest phases of the project life cycle have the most dramatic impact upon the odds of project success. The paper draws on the guidance in the PMBOK® Guide and from respected authors about these early project phases. It characterizes the importance of the project stakeholders to develop further the point that the future of successful project management involves a disciplined assessment of the opportunity. Through the introduction of some interesting spy projects, the author draws on experience from his twenty-seven years in the project management business in the CIA to describe examples of competing project choices.
Following the characterization of the issues and consequences associated with the earliest project phases, the paper describes a process recently adopted within a leading CIA information technology organization to improve their approach to identifying the right projects. Described is the Opportunity Assessment phase of their project life cycle, a phase that precedes the decision regarding the initiation of a project. This description contains the steps, control gates, and deliverables that are associated with Opportunity Assessment. Included are meeting with the customer, reviewing resources, strategy checking, stakeholder identification, risk assessment, feasibility assessment, and planning for the next phase.
The introduction of Opportunity Assessment within this relatively young information technology organization in the CIA is in keeping with the evolution of its business model to reflect shifting customer needs. What has been a customer-driven organization is now even more so. The paper summarizes the major challenges and the lessons learned. Included are buy-in on the part of the project teams as well as management, schedule change, and process transition. The paper concludes with a summary of the positive reaction across the organization to the introduction of Opportunity Assessment. Opinions of project team members, and managers are summarized, as is the beneficial effect that this methodology has had on the project success rate and the organization's project management maturity level.
Project choices abound. Information technology companies are flooded with far more propositions than they have resources for. The auto industry selects just a few models from numerous concept car choices. The food and beverage industries have long lists of products with potential of which only a few will receive the massive investment needed to bring them successfully to market. The aerospace industry's research and development efforts continue to yield many more candidates for airplane designs than can profitably be produced. And yes, even the CIA finds itself overwhelmed with more project choices than the taxpayer can possibly afford. How can we make the right choices about which opportunities should be pursued as projects? How can we avoid investing our limited money and manpower in dead ends? How can we avoid creating another dot-bomb software program, manufacturing an ill-fated automotive Edsel, marketing a flop like New Coke, flying a niche jet like the Concord, or building spy devices that look good on paper but are impractical for use?
It Is Distracting
Pursuing opportunities that do not prove successful not only drains our resources but distract us from working on those projects that will be winners. And clearly, isn't that the reason for doing projects in the first place? Isn't our goal to deliver something that satisfies someone's needs? Wouldn't project management life be wonderful if we knew with great certainty that the project we were so diligently working on would be successful? Think about the tremendous level of motivation for the project team members if they knew that they were associated with ensured success. Think about how the cooperation from those outside our project team would skyrocket if they truly believed that what we were doing would work. Think about the skepticism that would melt away the next time we reviewed a project proposal if success was assured.
It Lowers the Success Rate
Unfortunately, life is not at all like this project management utopia. It is quite the opposite. Failure seems to be the rule, not the exception. Estimates of project failure rates vary, but not one of them is encouraging. CIOs surveyed by CSC Index estimate that half of the information technology projects they have initiated have failed (Kaplan 2001). A Standish Group survey found that a staggering 31% of projects are cancelled before they are ever completed (Standish 1995). Probably the most discouraging estimates come from Gartner Group, which claims that over 70% of network management projects fail to meet their original objectives, and Forrester Research, which reports that 75% of Internet-based service projects fail (Kaplan 2001). These sources offer consistent explanations for the high failure rates. Invariably, decisions made during the formative stages of the project are to blame. Some characterize it as unrealistic expectations, some a lack of user involvement, while others cite requirements related issues. Regardless of the specific reason, the failures are consistently attributed to the early project phases. So there it is: lots of project choices combined with high failure rates. This is an unwelcome combination.
Exhibit 1. Early Projects
Exhibit 2. 1960‘s Project
Projects have been a constant throughout the history of the CIA, and picking the right projects has been of high importance. During the earliest days the OSS invented weapons and gadgets including limpet mines, specialized boats, and explosive powder (Warner 2000). During the 60s and 70s huge projects were undertaken to build imaging satellites and even to raise a sunken Soviet nuclear submarine (McCarthy 1999; Burleson 1979). And most recently, information technology projects have been undertaken to provide the capability to conduct computer searches in languages not understood by the user and to convert automatically the audio from radio and television broadcasts into text (Zakaria 2001). This is a rich and diverse history, not unlike so many other companies and organizations, where choices need to be made between competing opportunities. Furthermore, this is a history where investment dollars need to be targeted at the prospects with the highest payoff, where failures not only represent loss of image or profit, but also sometimes loss of lives as well.
Exhibit 3. 1970's Project
The Solution for the Future
Doing the Right Projects
The future of successful project management involves doing the right projects—not just doing projects right. That means making good decisions about which opportunities should become projects. In turn, that means examining the project life cycle. In some project life cycles or methodologies these decisions occur during the initial phase; other life cycles refer to this period as the feasibility phase, while still others refer to it as the concept phase. In terms of the PMBOK® Guide, this means focusing attention on the scope-related processes in the planning phase. Scope initiation and scope planning are specifically identified in the PMBOK® Guide as processes by which decisions are made about doing the right projects. Input such as the organization's strategic plan, the project selection criteria, and the associated constraints are considered during the decision-making process about whether the candidate opportunity is selected to become an actual project. Certainly a case can be made to support the point that making good decisions about which opportunities become projects is not a new consideration. Perhaps what is new, however, is the importance of this consideration for the project manager. A decade ago leading authors stressed the importance of projects being driven by requirements (Kerzner 1989) and making needs determination the first step upon which the remaining life cycle was based (Frame 1987). A decade later these same authors are stressing the need to do important work even before focusing on the requirements (Kerzner 2001) and they are stressing the importance of making the decision about which opportunities become projects (Frame 1994). The emphasis is now to base these decisions upon strategic planning and stakeholder interests. This is underscored by a redefinition of the role of the project manager to now include responsibility for these opportunity decisions.
The Stakeholders Are Important
The PMBOK® Guide makes a strong case, which is supported by current literature, for stakeholder involvement, particularly as it relates to making the decisions about doing the right projects. The stakeholder mix will most likely be situational, varying by organization and by project. For some the mix will include the project manager, customer, performing organization, and sponsor. For others the stakeholder mix may include owners and investors, suppliers and contractors, team members and their families, government agencies and media outlets, individual citizens, lobbying organizations, and society at large. How do we ensure that we are doing the right projects; we do this by making good decisions about which opportunities become projects. And how do we make good decisions about which opportunities become projects; we accomplish this by basing these decisions heavily on the stakeholder needs and expectations.
A leading CIA information technology organization has developed a structured and disciplined process, Opportunity Assessment, that is based on the best practices of evaluating project prospects; stakeholder needs and expectations are at its foundation. Opportunity Assessment has been formally inserted as a phase in the beginning of the life cycle. Project teams are following Opportunity Assessment to guide their efforts to select from the myriad of opportunities, while line managers and others follow this process to guide their interactions with the project teams. Mutual expectations exist about what needs to be done, how it will be done, and who will do it. All have received hardcopies of this process, all have received templates for the associated documentation, all have access to it as an online tool, and all have participated in training regarding its use. It details a common way for those involved to progress to the point of making a decision about whether the opportunity should become a project.
Opportunity Assessment is based on following a series of steps, which are sequenced, between control gates. Decisions and written deliverables are required to proceed past a control gate. Opportunity Assessment is the first phase of the life cycle; it begins when a lead from a customer is identified as a potential opportunity and enough interest exists to warrant investigating it further as a potential project. It ends with a decision about proceeding to the following phases of the life cycle. An Opportunity Review Board is convened to perform the control gate functions. All Opportunity Assessment steps, control gates, and deliverables are tailored to the particular situation; overlap between them is permitted. Some opportunities actually proceed through Opportunity Assessment in a day or two, while others take several months.
Exhibit 4. Punch Line
Exhibit 5. Current Projects
Opportunity Assessment begins with a control gate to make the decision to pursue a lead as an opportunity and with the assignment of someone to be in charge of the opportunity: an Opportunity Assessment Manager. Critical to this beginning is the need to make a reasoned decision and to assign a resource, both of which are essential if the undertaking is to become more than an accidental project. The Opportunity Assessment Manager performs an initial opportunity assessment by conducting a high-level resource review to assess whether the organization has the capacity for this undertaking; then identifies subject matter experts in the specific areas represented by the opportunity. At this point both the customers and the stakeholders are identified and interviewed. Their needs and wants are captured in writing to be used later in the process. The circle of people and organizations included in these interviews is typically broad and inclusive of those that currently are or in the future are expected to be in positions to exert influence over the project. Interviews are often conducted with actual end users, their management, the primary provider for their existing suite of information services, the organization responsible for providing operations and maintenance support, the management responsible for overseeing the assessment of the opportunity, and the policy makers. On occasion interviews conducted for one opportunity can, if still current, be used for a similar or related opportunity, thereby reducing the number of interviews. Checks are done to determine the fit of the opportunity to existing strategic plans and to determine its technical feasibility. A Statement of Need document is written and used as the basis for a control gate to conduct an initial review of the opportunity.
Exhibit 6. Opportunity Assessment
Opportunity Assessment continues with a check within the organization to see if an existing tool or previously developed solution could satisfy the need. Risks and mitigation strategies are identified and are likely to address customer/market uncertainty, technology dependence, personnel skills and availability, and funding and schedule availability. The Opportunity Assessment Manager evaluates the opportunity's feasibility by integrating the assessment factors into a table weighted by criteria for importance and compliance. An independent confirmation is made of the customer's commitment as well as the impact to the customer by satisfying the need; if necessary, revisions are made to the information obtained from the previous steps. An Opportunity Assessment Report is written and used as the basis for a control gate to conduct a final review of the opportunity.
Opportunity Assessment concludes with the planning and resource allocation necessary for the next phase of the life cycle. At this point the resource needs typically begin to expand somewhat to include people with technical skills in the subject area. An Opportunity Planning Manager is identified; this may be the same person as the previous Opportunity Assessment Manager, and/or it may be the same person as the eventual Project Manager. An Opportunity Plan, which includes a schedule, is written and used as the basis for a control gate to conduct a review of the plan for the next phase.
The Opportunity Review Board, which performs the control gate functions, is comprised of members who represent the key customers and stakeholders. Also represented is the management responsible for overseeing the assessment of the opportunity. Board members make decisions based on the information made available by the Opportunity Assessment Manager as well as information they have available through their representative functions. They also perform a quality control function by ensuring that the prescribed process for Opportunity Assessment is followed. Information flow is invariably two ways with both the board members and the Opportunity Assessment Manager benefiting from the exchange at each control gate.
The organizational elements within the CIA that are using Opportunity Assessment are using it as part of a total life cycle that was formally adopted during the last couple of years. The development and adoption of this formal process was driven from throughout the organization by the desire of the project teams for procedural consistency and by the organizational management's desire for more predictable project results. It was requirements-driven. While this underpinning of broad support did have the effect of minimizing the challenges associated with implementation, some hurdles did arise. Midway through the process development, a request was made to speed up the schedule so that the process could be in place coincident with the beginning of the fiscal year, a time associated with the beginning of many projects. A consequence of this shortened schedule was a need to reduce the scope of the effort by decreasing the number of templates for the documentation associated with the process; those templates were delivered at a later date. When the process adoption was about to become official policy, a few projects were given additional time to complete their preparation.
Exhibit 7. Life Cycle
Shortly after Opportunity Assessment and the associated total life cycle were adopted, the major contributors developed a list of lessons learned. The consensus was that the effort to develop and adopt this formal process was successful not by accident but by intentional actions. Key lessons included the fact that early on a senior-level manager served as a sponsor for the effort and maintained a consistent focus on it. Equally important was the fact that the effort was treated as a project; a dedicated, experienced project manager was assigned who had the discipline to manage the tasks and resources according to an approved plan, and a cross-organizational team served as the governing body. Other lessons learned included good working dynamics among the team members, a shared corporate viewpoint, and a sincere belief by the participants that the project scope and goals were attainable. Perhaps the most important lesson learned was that the project was requirements-driven; people throughout the organization genuinely wanted it.
Young leading edge organizations that introduce process standards often do so as part of an ongoing strategy focused on continuous improvement. This has been the case for Opportunity Assessment. What was being done well is now being done better, and the benefits of using it continue. As part of the overall life-cycle process it has already been found by the project teams to reduce the uncertainty associated with getting projects started. It has also helped to make the efforts of project teams more consistent with each other, while simultaneously increasing clarity for the management reviews of the projects. Both project team members and management speak openly about the benefits and actively work to expand its application to more and more projects. Project management maturity has been measured and found to be greater for the organizational elements where Opportunity Assessment has been used for longer periods of time. And some higher-risk opportunities that might have become projects in the past are being avoided; as a result, project success rates that were already good are now even better. As for the bottom-line objective of making sure the organization is doing the right projects, Opportunity Assessment is obviously helping.
Burleson, Clyde. 1979. The Jennifer Project. Sphere Books Limited.
Frame, J. Davidson. 1987. Managing Projects In Organizations. Jossey-Bass.
Frame, J. Davidson. 1994. The New Project Management. Jossey-Bass.
Kaplan, Jeffrey. 1998. Raising Your Network Management Batting Average. NetKnowledge. Lucent Technologies.
Kerzner, Harold. 1989. Project Management A Systems Approach to Planning, Scheduling, and Controlling. Van Nostrand Reinhold.
Kerzner, Harold. 2001. Strategic Planning For Project Management Using A Project Management Maturity Model. John Wiley & Sons.
McCarthy, Linda. 1999. Spies, Pop Flies, and French Fries. Markham, VA: History as a Hoot, Inc.
PMI Standards Committee. 2000. A Guide to the Project Management Body of Knowledge (PMBOK® Guide). Upper Darby, PA: Project Management Institute.
The Standish Group. 1995. The Chaos Report. West Yarmouth, MA: The Standish Group.
Warner, Michael. 2000. The Office of Strategic Services: America's First Intelligence Agency. Washington DC: CIA Office of Public Affairs.
Zakaria, Tabassum. 2001, March 2. CIA Using ‘Data Mining’ Technology To Find Nuggets. Reuters.
Proceedings of the Project Management Institute Annual Seminars & Symposium
November 1–10, 2001 • Nashville, Tenn., USA