Project management certification now underway at the CIA
The Central Intelligence Agency (CIA) is not only writing a new chapter in its management of projects book, but it is also is writing an entirely new edition. The previous edition, which has served us well, was acquisition and life cycle focused and gave particular emphasis to the larger and longer technology-based projects, of which we've had many. However, numerous employees found it a difficult read with the result being that its application was spotty, and even those who did apply it found hurdles associated with its tailoring and use beyond a somewhat narrow set of circumstances. This new edition has been crafted to address the shift in the nature of the projects in the CIA and that of the project managers as well. It is being authored to be applicable to a broader range of CIA project types and an equally broad mix of project managers. It is raising the bar within the CIA for the practice of the profession of project management.
This paper addresses the aspects of the comprehensive agency-wide Project Management Training and Certification Program (PMTCP) that have transpired since its launch less than a year ago. The key features of this multi-level, A Guide to the Project Management Body of Knowledge (PMBOK® Guide) aligned, test-based program are summarized and are contrasted with what had previously existed; contrasts include the shift to the broad profession of project management from the narrower areas of acquisition and procurement, to increased student application of key skills in addition to basic awareness, and to industry best project management practices instead of just those unique within the CIA. Described are the extensive efforts to market this program to the work force, including working groups, a multifaceted communications campaign, and the unexpectedly large interest in pilot participation. Emphasis is given to the events surrounding the rigorous development process used for the dozen courses and associated tests and the related effects of this not only on the course developers but also on the pilot students as well.
The full curriculum of project management training that is being developed to support the certification is described. The status of the PMTCP program is presented and is followed by an explanation of the key reasons for the early success, including the use of a clear-cut set of course development standards and lesson plans. The paper concludes with a summary of the next steps for the program as it moves from pilot phase to implementation across the enterprise.
The CIA's long rich history involving thousands of projects has positioned the organization to respond to the genuine interest by project managers and by executives to raise the bar for project management (CIA 2003). This interest is being driven by change: CIA projects are following trends toward smaller size and shorter life cycles, CIA demographics are following trends toward a younger and less experienced work force, an awareness is growing of the tried-and-true best practices being adopted outside of the CIA, and project management is being applied more broadly to work other than traditional development. Key individuals within the CIA, who recognize this need for change, have seized the opportunity to create an agency-wide project management training and certification program. This program has been created by project managers for project managers. It integrates project management and systems engineering, remains faithful to our standard project life cycle methodologies and vocabulary, aligns with the PMBOK® Guide, and bases certification on testing. This integration is captured in the PMTCP documentation of requirements, concept of operations, and statements of work. Competitive contract awards were made to two leading project management training companies to adapt their existing mature material to our PMTCP needs, to develop with the assistance of a leading testing company the bank of tests necessary for certification, and to integrate this into a comprehensive program (O'Brochta, 2003).
This PMTCP program was planned and is being managed following a tailored version of one of our standard project life cycle methodologies. Key sequential tasks include requirements definition, planning, acquisition, requirements validation, development, piloting, transitioning, and implementation. Key parallel tasks include the management of the project and communication. Key control gates include Requirements Review, Design Concept Review, Pilot Readiness Review, and Annual Readiness Reviews. At this point most of the courses and tests have been piloted, demand has been very high, and student feedback has been very good. Course and test revisions are underway in time for open agency-wide enrollment and transition to a full production implementation (Exhibit 1).
Exhibit 1. Plan
Professional Project Manager Certification (PPMC)
At the core of the Project Management Training and Certification Program at the CIA is a set of thirteen courses and tests. These are structured into four hierarchical levels that are aligned with the grouping of needs from both the project management population as well as from the senior executive population. These hierarchical levels correspond to increasing levels of project schedule rigidity, complexity, and dependencies (Exhibit 2). All but one of the PMBOK® Guide knowledge areas are covered, as are areas associated with systems engineering and general business management. The omitted PMBOK® Guide area of procurement is covered by a companion certification at the CIA for employees involved with acquisition who serve as Contracting Officer Technical Representatives (COTR's).
Exhibit 2. Hierarchy
The first certification level is intended to provide overall familiarity with the responsibilities of project managers and the common terminology associated with the field. A single 5-day course equips project managers with the knowledge and skills to work as team members on large projects or as leaders of small low-risk projects. It is assumed that project managers at this certification level will receive significant supervision.
The second certification level provides a depth of knowledge and skills across the field of project management as it is practiced in the CIA. A series of six 2-4 day courses focuses on knowledge and skills associated with the management of project integration and scope, time, cost, risk, communications, and the project team. Certification at this level prepares project managers to perform all functions for low-risk to moderate-risk projects. This level of certification contains much the same content as the Project Management Professional (PMP®) certification available from the Project Management Institute (PMI®).
The third certification level is focused on the systems engineering aspects of project management. This is being accomplished in a series of four 2-4 day courses that concentrates on knowledge and skills in the areas of systems engineering principles, requirements development, concepts and architecture development, and integration, verification and validation. Project quality management is addressed throughout this level as well as in some of the level two courses. Certification at this level prepares project managers to perform all functions for moderate to high-risk projects.
At certification level four the focus is on producing project management experts capable of leading significant and high-risk projects. Two 5-day courses equip project managers with the knowledge and skills in the areas associated with complex and strategic projects and programs. Project managers at this level are expected to work outside of the routine organizational and procedural bounds to mature the project management culture.
Supplementing this core set of thirteen courses and tests are opportunities for employees to take certification tests without going to class. These opportunities are particularly well suited to those employees who feel confident in the knowledge and skills they have gained through experience and/or self-study. These employees can choose among the same Post-Course tests associated with each of the thirteen courses. They can also choose to take a comprehensive Diagnostic/Equivalency test, which covers each of the four levels of certification; employees who take this test also receive feedback regarding their areas of strength and opportunities for improvement.
Certification is granted by a single CIA authority and is based solely on test results. This internal CIA certification, which is called the Professional Project Manager Certification (PPMC), is awarded at each of the four progressively higher levels. “Grandfathering,” whereby practitioners are granted certification due to their seniority or experience level, is not permitted. Tests are multiple-choice pass/fail; cut scores are not publicized, and test results are kept confidential. Tests typically involve 50 questions, are not timed, and with the exception of level four are closed book. Test questions are written to engage the student in a process of thought to analyze the presented situation and to apply their knowledge and skills in the process of selecting the correct answer. The test questions can be characterized as “hard but fair.”
Prerequisites are not required and employees may take the courses and tests in any sequence. However, since some of the courses do build sequentially employees are encouraged to take them in order. Certification is only awarded sequentially. This means, for example, that an employee who passes all of the tests associated with level two certification will receive that certification only after level one is passed as well. Employees may take the courses and tests at whatever pace is suited to their aptitude and interests, and fits with the work schedule established by them and their superiors. Someone progressing through all thirteen courses at all four levels will be engaged in up to 42 days of classroom training likely spread over a 3-5 year period (Exhibit 3).
Exhibit 3. PMTCP Courses
PMTCP represents a significant shift in the approach to project management in the CIA (Exhibit 4). At the heart of this shift is the recognition that the nature of the projects underway at the CIA has changed and continues to change. The definition of the term “project” has expanded to encompass not only the traditional high-technology procurement contracts, but also an increasing number of smaller shorter-duration low technology in-house managed efforts. Project lengths of one year or less are more the norm. Projects where incremental deliveries are desired now represent the norm. Projects employing rapidly changing technologies are more the norm. The strict following of highly standardized process-driven life cycles are giving way to the use of a deeper knowledge and skill set to do what is best for each individual project.
Exhibit 4. Changes
The CIA's Directorate of Science and Technology recognizes this shift and has resumed its position as lead for the changes necessary to capitalize on them (DST, 2003). The responsibility for agency-wide project management training has now returned to this Directorate from its temporary home elsewhere in the CIA. Emphasis is being given to project management as a separate and distinct formal occupation instead of “other duties as assigned.” In addition to learning “what” project management actions need to be done, employees are now learning “how” to do them. For example, many more of our project managers are finding that their skills at reviewing a work breakdown structure (WBS), traditionally written by a contractor, is no longer adequate; they are finding that since they are not relying as much on the contractor, they need to be able to create a WBS themselves. Similar circumstances exist for other knowledge areas as well. The emphasis for training our project managers is shifting away from exposing them to a large amount of information to having them actually develop and retain the knowledge and skills associated with a smaller key subset of those topics. The topics are changing too; whereas the life cycle used to be on the cover of CIA's project management book, knowledge areas have replaced it. The end goal here is to employ project managers who have the depth of knowledge and skills to select, tailor, and effectively use a variety of life cycles depending upon the situational needs of their project. We are broadening our view of project management as well. Our years of repeatedly reusing internal CIA project management practices are giving way to a rapidly growing awareness and use of the standard for project management, the PMBOK® Guide.
Interestingly, this significant shift that is represented by PMTCP is optional for virtually all project managers. With the exception of some of the more senior project managers who are also COTRs for the larger contracts, project management employees may or may not choose to seek this internal agency certification. Those who value the increase in their own knowledge and skills are gravitating toward obtaining this certification as are those that consider having this certification to be a competitive advantage when assignment and advancement decisions are made. Those who do try for this certification encounter the very new reality of having to pass a test. For most CIA employees this test dynamic is new, and it replaces the old dynamic where credit was obtained for merely attending class, for experience, or for seniority. This test dynamic is causing some fascinating effects. For the course developers it imposed a constraint to establish and adhere to a set of Lesson Training Objectives (LTOs) for each course; this, in turn, limited the developer's freedom to make course changes. It also fostered a development discipline that yielded a high level of quality. These LTOs served not only as the basis for the course development but also for the test development as well. For the students, it raised their level of commitment to attending and participating in class, as well as to their actually learning the material. Student attendance problems and dropouts have been virtually eliminated, and homework assignments are met with enthusiasm.
How We Got This Far
Some of the common organizational barriers, such as the lack of resources, the lack of top-level management commitment, the lack of a clear solution path, or the lack of patience to do the job, have not been significant factors. Enough of the top-level managers are acting on their support for the creation of PMTCP by providing the necessary resources. Enough knowledgeable voices are aware of the highest payoff solution paths and are involved to the degree necessary. And, enough key stakeholders exist who are patient enough to spend the time necessary to achieve a high level of quality. Resistance to change, particularly on the part of the more experienced project managers, is another all too common barrier. This could have been particularly true at the CIA given the large installed base of trained project managers and mid-level managers who have predominantly known only one form of project management. Yet, even this most significant barrier has not proved to be much of an obstacle. Those who have voiced an opinion have revealed their surprise at finding so much of value in PMTCP, using clichés such as actually “teaching old dogs new tricks.” There are likely some naysayers who we have not heard from yet, but addressing their concerns will likely be easier given the ever-growing numbers of committed supporters.
Keys To Getting This Far
The “raising the bar” theme has worked well, both as an operating principle and as a marketing strategy (Exhibit 5). Both course developers and prospective students have responded to the challenge of being associated with higher levels of quality and performance. From this we have built and used industrial strength lesson plans that are compliant with a PMTCP Standards Document. Learning objectives based on Bloom's taxonomy precede all of the actual development (Bloom, 1956). Both the form and the content of the courses and tests actually were driven by the needs of the project managers. An effective communications program was established with these project managers and used both as a source for the requirements and as a mechanism for vetting the evolving course designs; the cornerstone of this effort was the creation of a Project Management Standards Working Group (PMSWG). This working group is comprised entirely of motivated project management volunteers, some of whom are working at the practitioner level and some of whom are working as first-and mid-level managers. The PMSWG participated in key control gates during the development process, participated in the pilots as students, and served as leaders within their local components for the transitioning to the work force. This working group was directly responsible for the lowering of many of the anticipated barriers; they developed the Professional Project Manager Certification (PPMC) brand that has effectively captured the attention of the work force, and they helped develop the agency-wide project management policy that regulates the program's implementation (Exhibit 6).
Exhibit 5. Keys To Success
The communications and marketing of this program to the work force occurred at numerous levels. Target audiences were defined and communications programs were tailored to their needs. Key senior executives received one-on-one personalized visits that allowed them to understand the benefit and impact to them and their work force (Thomas, 2002). The value to project managers and line managers was quantified and disseminated via briefings to organizational work groups (Pennypacker, 2002; Ibbs, 2002). Course descriptions and policy notices were posted on an agency-wide website, and they were discussed at an all-day kickoff conference given to a 400 person capacity crowd in the CIA Headquarters auditorium. As a result, enrollment for every pilot course exceeded the prescribed twenty-four-person capacity and waiting lists grew to almost as long. These pilots were well instrumented, with evaluations produced by the students, by the instructors, and by professional course evaluators. Students who did participate in the pilot courses and tests did invest extra time and effort performing these evaluations, and they did experience course and test material with occasional rough spots. Consequently, all pilot participants received bonus points worth ten percent on their tests, and they received gift certificates redeemable at local restaurants.
Exhibit 6. Logo
The content of the program itself has relieved some of the anticipated barriers. PMTCP builds on instead of replaces previous approaches to project management and it broadly targets the many different types of CIA projects. The content remains faithful to CIA terminology and life cycles while introducing new PMBOK® Guide terms. It maintains the CIA integration of project management and systems engineering while introducing new PMBOK® Guide knowledge areas. And this content is being delivered in the classroom by co-instructional teams, comprised of an experienced contractor instructor as well as an experienced CIA staff instructor. These co-instructional teams are able to minimize the potential gap between the project management theory and its application to the CIA types of projects; these teams raise the relevance level of the material.
Here too, the test dynamic has played a role. It has underscored the seriousness of the program as a legitimate and bonafide accomplishment, and it has helped draw attention to the message that certification is granted using a fair and unbiased process. It distinguishes the program by setting it apart from the volume of training in which testing is not required.
The teams working on PMTCP are good at what they do, and they appreciate the fact that PMTCP is being managed as a project complete with its own control gates and deliverables. The contractor course and test developers are experienced professionals who have the discipline to deliver top quality products (CSM, 2004; MCI, 2004; Prometric, 2004). They actively solicit involvement from members of the PMTCP program office, and they use the information and guidance they receive. The CIA staff is experienced at project management and instructional delivery, and they know what will work well with the target audience and what won't. The commitment levels of the contractors and the CIA employees working on PMTCP are high. This is equally true for the PMTCP program manager, who demonstrates a high level of motivation and vision as a leader by supporting other team members, and who demonstrates his level of commitment by direct involvement in much of the actual development (Exhibit 7).
Exhibit 7. Punch Line
What We Will Do Different Next Time
Although we are not entirely finished with the development and piloting of the courses and tests, and although we have had only hundreds and not yet the expected thousands of students, we have gained considerable insight into how we would do things differently next time. At the top of this list would be more time. The planned length of the development schedule was six months; a year would be more realistic. Although approximately eighty percent of the course material is off-the-shelf, it is still taking considerable effort to repackage it into courses customized to meet our requirements. In particular, developing level four material is proving to be somewhat elusive, involving considerably more than just repackaging. We'd also plan on more effort for the development of the test questions. As it is turning out, fully one-third of the total cost to develop and pilot PMTCP is attributable to the tests. Admittedly, we raised the bar for the tests just as we did for the course material. These are very high quality tests, capable of accurately assessing who falls above the competency threshold and who does not. They are capable of withstanding scrutiny by occupation and human resource specialists, and capable of withstanding legal challenges. Thankfully, we are working with an extremely competent test development company, one with an abundance of experience developing these sorts of tests. But, as we have learned, the rigorous discipline of this type of test development takes time and costs money.
This test dynamic would also cause us to more tightly couple the linkage between the course content and the actual wording of the test questions. In some cases this linkage was lost during the development and revision process causing some pilot students to be exposed to test questions about material that was not fully covered in class. We would also add another control gate to the course development process. The inclusion of a Critical Design Review, following the early design efforts and reviews and preceding the actual development of the student material, would have allowed decisions to be made based on a thorough review of lesson plans and course maps. This additional review would have likely prevented us from the piloting of one course with unacceptably low quality and would have saved us the trouble of undertaking significant redesign and re-piloting. Similarly, we would plan next time for even more involvement between our subject matter experts (SMEs) and the course developers. These SMEs played a pivotal role insofar as their ability to guide the setting of the context of the student material for our work force.
Virtually every course we have developed has had too much material for the time available. In most cases this fact was discovered during the latter stages of development or at a pre-pilot Readiness Review. In a few cases it became apparent only after the pilot was underway. Our course development Standards Document, which was built specifically for PMTCP, did contain many items of considerable value to the course developers. However, it was silent in areas related to timing. We've since amended it to include standards for slide density, slide pacing, and number of training objectives per day.
What Is Next
Finishing the pilot courses and tests and keeping up with demand will occupy most of the attention for the next year. Four of the thirteen courses and tests remain to be piloted. Since a large demand has been projected, classes have been scheduled to offer: the single level one course twelve times during the next year, all six level two courses six times during the year, all four level three courses four times, and both level two courses twice during the year. Post-Course tests are being offered at the end of every course as well as twenty-four other times, and Diagnostic/Equivalency tests are offered a dozen times.
A production engine is being developed to support these 68 classes. Configuration management procedures will be written and followed. Biannual change control and a configuration library will be established and maintained. Standard sets of operating procedures will be developed and followed for the administration of PMTCP including enrollment, instructor assignments, course material delivery, registration, and test proctoring. The design of a metrics program capable of assessing PMTCP impact to the individual student and to the organization will be completed and implemented. An adjunct instructor program will be established and requirements for recertification will be developed.
Depending on funding, supplemental activities may begin. Under consideration are facilitated workshops where students are guided toward achieving specific project management aspects of their actual projects; project launch, project planning, project risk assessment, and project takeover/turnaround are candidate workshop topics. Also under consideration are consulting services, such as project management and executive mentoring, project planning, and organizational maturity assessments. At some point the knowledge delivery model reflected in PMTCP will shift sideways, with structured and supported opportunities for horizontal knowledge sharing directly between project management practitioners instead of only the vertical path between the experts and the practitioners.
Bloom, B., Englehart, M. Furst, E., Hill, W., & Krathwohl, D. (1956) Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitave Domain. New York, Toronto: Longmans, Green.
Center for Systems Management (CSM), 2003. Professional systems engineering certificate. www.csm.com/Pages/SitePage-Se-Certificate.shtml.
Central Intelligence Agency (CIA), 2003. Factbook on intelligence. www.cia.gov/cia/publications/facttell/index.html.
Directorate of Science and Technology (DST), 2003. The directorate of science and yechnology: An overview of the first forty years. Central Intelligence Agency (CIA):Washington D.C.
Ibbs, William. (2002) Quantifying the value of project management. Newtown Square, PA: Project Management Institute.
Management Concepts Incorporated (MCI), 2003. Project management resource catalog. www.mgmtconcepts.com/scripts/mcibrowse.asp?Catalog=PJ.
O'Brochta, Michael. (2003, October) Launching a project management certification program at the CIA. PMI Global Congress 2003, Baltimore, Maryland.
Pennypacker, James. (2002) Justifying the value of project management. Haverton, PA: Center For Business Practices.
Project Management Institute. (2000) A guide to the project management body of knowledge (PMBOK® Guide). Newtown Square, PA: Project Management Institute.
Prometric, 2003. Global IT training and certification study results. www.prometric.com/PressRoom/GlobalITStudy.htm.
Thomas, J., Delisle, C., Jugdev, K. (2002) Selling project management to senior executives. Newtown Square, PA: Project Management Institute.
© 2004, Michael O'Brochta, PMP
PMI Global Congress Proceedings • October 21-28, 2004 • Anaheim, California, USA