Content validation for the Project Management Professional (PMP)® certification examination

THE PROJECT MANAGEMENT INSTITUTE'S Project Management Professional (PMP^®) certification examination, in addition to education and experience, supplies the framework for awarding the highly valued credential to individuals who demonstrate compliance with PMI^® certification requirements. PMI invests considerable effort to ensure the quality of the examination. One of the most important qualities of a certification examination is its validity. As part of its ongoing attention to quality, PMI undertook an important series of activities in 1999 and 2000 to make certain that the PMP examination is a current, valid, and appropriate measure of professional knowledge in project management. These activities were driven by three essential questions:

What is the role and purpose of assessment in a certification program?

What is the purpose of the Project Management Professional examination?

How should PMI ensure the examination's validity?

After addressing these questions in light of an analysis of the PMP program and accepted principles in the fields of testing and measurement and professional certification, PMI undertook a Role Delineation Study. I'll discuss PMI's investigation into the three essential questions, PMI's resulting activities, and then look at what a Role Delineation Study is, and why it is the correct validation strategy for the PMP examination.

What is the Role and Purpose of Assessment in a Certification Program? Assessment is a process that takes many forms, depending on its purpose, and may involve many tools, including examinations: personality assessments, intelligence tests, achievement tests, placement tests, credentialing tests, and so on. The purpose of an assessment—the conclusions that will be made based on scores or other products of the assessment—provides the foundation for decisions about the content of questions or problems, the nature of the questions or problems, as well as the analytical strategies employed to determine if the test achieves its stated purpose. If there is evidence that an assessment supports the intended decision-making, it can be said to have validity.

Understanding the types of decisions to be made on the basis of scores is critical in evaluating if a test is useful for its intended purpose. In educational systems, for instance, assessment provides useful information about the progress and current achievement of the learner. That information can then be employed to enhance curriculum development, place the learner in the most appropriate environment, or determine summatively if an individual or group has satisfied program objectives. Every aspect of test development for assessments that serve educational purposes is aimed at maximizing the ability of the test to support the educational decisions to be made on the basis of scores. Questions may be selected as a representative sample of the relevant curriculum, questions may be selected to achieve the broadest possible range of difficulty, and scores may be interpreted in relation to a norming group, often with scale comparisons that permit diagnosis of strengths and weakness.

In certification programs, the assessment process is designed to make an either/or decision: either to certify an individual or not to certify. Assessment devices in certification programs are not usually used to diagnose individual strengths and weaknesses, to discriminate between very good and excellent ability levels, or to improve professional educational programs. Instead, they are criterion-referenced through linkages to a standard of knowledge related to the defined acceptable level of professional service. Every aspect of test development is geared to support the specific certification decision-making purpose. For instance, questions are evaluated relative to their importance and relevance to practice (not necessarily what has been taught in training programs) and the degree to which adverse consequences may result if the certified professional is unknowledgeable in the content area addressed by the questions. Scores are compared only to the passing standard and no quota is used to determine how many individuals pass the test.

Reader Service Number 054

What Is the Purpose of the PMP Examination? The PMP certification identifies project management professionals who have satisfied PMI's standards for training and experience and who have demonstrated the required level of knowledge about project management by passing the PMP examination. The only decision made on the basis of scores is whether or not individuals are awarded the PMP designation.

PMP certification has value for the project management professionals who possess it, as it is an important basis for employment decisions, selection of contractors, and advancement within project management. In using the services of a PMP, employers and members of the public (however broadly this term may be defined) rely on the credential as an indication that the PMP possesses the ability to carry out the essential functions of project management in accordance with accepted practice. The PMP examination plays a critical part in the certification program in that it provides the standardized benchmark against which candidates for the PMP designation are assessed relative to a common standard of essential practice-based knowledge. Because of its purpose and value, the PMP examination may be classified as a high-stakes examination.

Because of the nature of the examination, PMI adheres to a system of principles and standards that are defined by the certification and testing and measurement communities to ensure that the interests of employers, other members of the public, and candidates are adequately served.

Employers and other members of the public require assurance that program requirements are sufficiently rigorous to justify the claim that PMPs are capable of conducting project management activities in accordance with accepted practice. Candidates also expect rigor, but have additional expectations concerning equal treatment and fairness. PMI demonstrates its commitment to these important obligations by seeking expert consultation in all areas of the certification program and external verification of its compliance with accepted principles and standards.

As part of a high-stakes program, the PMP examination must be valid with respect to the responsibilities of project management and the knowledge required to carry out these responsibilities. Validity refers to the degree to which the examination serves the purposes for which it is intended; that is, to distinguish which individuals possess required knowledge of accepted project management practices and which do not. For a high-stakes professional examination, the fundamental requirement for validation is that the content of the examination bears a demonstrable linkage to professional practice.

How Should PMI Ensure the Examination's Validity? Evidence of content validity takes many forms. There must be a compendium of accepted project management practices and the essential knowledge associated with those practices. There must be evidence that members of the profession have reviewed the compendium and agree that it accurately depicts professional practice and knowledge. The examination must be shown to emphasize the elements of professional practice and knowledge as described in the compendium according to their relative importance to accepted practice. Each question must be written, edited, and reviewed by qualified professionals. The questions must conform to standards for importance and relevance as well as form and quality. Published literature must verify the accuracy of each question. If each of these requirements has been satisfied (in addition to those concerning establishment of the score required to pass the examination and statistical evidence that the examination functions appropriately), then it may be said that there is persuasive evidence for the content validity of the examination.

Each year PMI conducts international meetings with experts in project management to prepare, review, edit, reference, and validate new and existing questions for its test in order to satisfy the requirement for content validity. PMI conducts additional meetings with experts to review the performance of questions after they have been used on the examination, and to assemble new versions of the test that are comparable in content and difficulty. A number of textbook reference sources are used by PMPs during the item development phase of the test development process. The acceptable questions from this phase are then reviewed by PMPs during the validation phase. In addition to validating the psychometric integrity for each question, the correct answer and plausible distracters, the validation process ensures that the examination content is consistent with how project management knowledge is described in the PMBOK^® Guide.

PMI regularly conducts statistical studies concerning the quality and function of the examination and uses this information to update the examination and to ensure conformity to accepted principles and standards. Most fundamental to this process, however, is the recent Role Delineation Study that PMI conducted in 1999 and completed in early 2000. The role delineation study is the means by which PMI developed and validated the compendium of responsibilities and the knowledge base associated with each responsibility. The present study updates one completed in 1997.

What is a Role Delineation Study? Why Is It the Correct Validation Strategy for the PMP Examination? A role delineation study is a widely used method for identifying the function, or role, played by professionals who are responsible for a particular range of professional practices. Role delineation study is one of a group of methods similar to job analysis. The purpose of a role delineation study is to define and list the domains of responsibility that constitute the role. Normally there are two phases in conducting a role delineation study: First, develop a compendium that describes the domains of practice, tasks that constitute each domain, and the knowledge and skill required to perform the tasks. Second, collect evidence from practitioners on whether or not the compendium accurately describes accepted practice. A role delineation study can be distinguished from a job analysis in that a role is defined by the function of the professional even though there may be significant differences in the types of problems (or projects). With a wide range of problems that may be addressed, the individual in the role applies accepted practices to adapt to the unique circumstances of the setting, project, or practice.

In project management, the role transcends specific fields or disciplines. The role is defined by identifying general practices and strategies, or tasks, which are used by the professional according to the dictates of the unique environment or application (project). In contrast, a job is usually characterized by a clearly delimited set of tasks that are performed in the same or very similar manner because the differences in problems (or projects) are relatively small. There is usually a clearly understood product that results if a job is performed successfully. Individuals working in a job carry a title defined by the product of their efforts.

The focus of PMI's role delineation study is the process, or accepted practice, of project management. The focus on accepted practice is consistent with the idea that the PMP is responsible for fulfilling a defined role. While it is desirable that projects be successful, the PMP may play the role quite capably and the project may still be unsuccessful due to circumstances beyond the control of the PMP. If certification implied that the professional filling a role will be successful with every project, product, patient, and so forth, then one would expect to evaluate oncologists on the cure rate for their cancer patients, teachers on their success in equipping every student with grade-level achievement, and financial planners on the growth in wealth of their clients. In fact, oncologists, teachers, and financial planners may follow best practice and still have unsuccessful results from their efforts because of events they cannot control. Like the practice of medicine, teaching, and financial planning, project management may indeed follow correct practice yet not achieve successful outcomes.

By contrast, individuals performing jobs like wiring a building, painting a house, or maintaining the financial records of a business are more reasonably evaluated on the outcome of their efforts. Certainly there are accepted practices and essential knowledge bases characterizing these professions, but they are implemented with greater uniformity, and their correct implementation will lead to successful outcomes. If the outcome of a job is not successful, the fault can normally be attributed to a flaw in the manner in which the professional performs (barring material defects).

Given the certification decision, PMI's obligations to employers, the public, and candidates for certification require that the PMP examination bear a demonstrable linkage to accepted practice, which for the PMP transcends specific fields or disciplines. There must be evidence that the examination assesses knowledge of practice that is, in the judgment of experts in project management, at the level required to satisfy PMI's obligation to the consumers of PMP services. Because PMPs are most reasonably evaluated on the correct practice of project management rather than on the success of the project, and because project management is an enterprise that transcends specific fields and disciplines, role delineation study is the appropriate approach to content validation for the PMP examination.

There are several ways of conceiving of knowledge related to the practice of project management. Under the taxonomy defined by Bloom, et al. [Bloom, Engelhart, Furst, Hill, Krathwohl, Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain, New York: David McKay Co.], the cognitive domain is characterized by five levels: recall and understanding, application, analysis, synthesis, and evaluation. Most examinations used for the purpose of certification emphasize the first three levels, with particular stress on application and analysis. The reason for the emphasis on application and analysis is that test questions at these levels require candidates not simply to recall or understand facts and principles but to employ them in addressing or solving practice-related problems. In this manner, it is reasonable to think that the assessment addresses professional competence in terms of knowledge, problem solving, and decision-making in matters commonly faced in the work setting. PMI has employed this framework in developing its examination. The role delineation study has pointed to areas where the examination is used to simulate professional problem solving and decision-making.

Many publications exist in project management. They are diverse in their scope and purpose. Some address issues and practices that are specific to certain types of projects, particular aspects of project management, or disciplines in which project management services are provided. Others are general sources aimed at identifying strategies linked to the success of projects. The diverse publications are available from many sources and represent many points of view. The extensive literature on project management is advantageous in that PMPs can draw on the experience and views of others to maintain currency with respect to trends and practices in project management. Although these resources, including the PMBOK^® Guide, make an important contribution to the body of knowledge that PMPs rely on in their work, their value is distinct from the contribution made by the role delineation study. While PMI uses this extensive literature in referencing and validating test questions, the role delineation study is specifically designed to describe the essential role of PMPs as the basis for content and context determination for the certification examination.

Reader Service Number 005

Overview of the Role Delineation Study. PMI conducted the role delineation study under the leadership of Columbia Assessment Services Inc. (CAS), a professional testing company specializing in the development of certification and licensure examinations. PMI convened an international and representative panel of project management experts in May 1999 for the purpose of defining the domains of practice, the essential tasks that constitute the domains, and the required knowledge and skill. Members of the panel were selected to achieve the broadest possible view of project management, with expertise in a variety of fields and disciplines. CAS facilitated the meeting in such a way that all viewpoints represented by the panel were considered.

The result of the meeting was a detailed compendium describing the six domains of project management (Initiation, Planning, Executing, Controlling, Closing, Professional Responsibility), the tasks that constitute each domain, and the knowledge and skill required to perform the tasks. PMI then subjected the compendium to international scrutiny by means of a validation research study in two steps. It first conducted pilot studies to assess the appropriateness of the sampling strategy and data collection process. Then it conducted the full-scale validation study. Nine hundred twelve randomly selected PMPs in the United States and 365 internationally were asked to participate in the study by completing a detailed survey, and 548 in the U.S. and 278 internationally chose to do so. Respondents estimated the degree to which (1) domains and tasks are important elements of the practice of project management, (2) domains and tasks are relevant to the practice of project management, and (3) adverse consequences might result in the event that PMPs did not or could not perform these practices. The results of this research enabled PMI to conclude that the compendium and the weights for different elements of the PMP examination derived from it are valid representations of the practice of project management. Consequently, PMI will use the role delineation study as the basis for the examination.

THE TYPES OF DECISIONS made on the basis of scores from an examination define the intended use of the assessment and guide the process of developing the examination so that it supports the decision-making process. Professional certification in project management indicates to employers and other members of the public that the PMP has satisfied specified requirements and passed a practice-related examination. The certification decision made by PMI is simply either to certify an individual or not to certify. In keeping with the certification decision and the validation requirements accepted in the testing and measurement and certification communities, PMI undertakes many activities to ensure the validity of the PMP examination for its intended purpose. Fundamental among these activities is the role delineation study completed in early 2000, selected from various options based on the nature of project management, which is made up of a set of practices that transcend industries, types of projects, and field or discipline. PMPs fulfill a role defined by the activities that constitute accepted practices rather than by the product or outcome of the activities.

By analyzing essential issues about the PMP examination program and following through with appropriate activities, PMI has developed a strong basis upon which the quality of PMP certification is assured. PMI will continue to undergird the quality of this program with research and development so that the relevant audiences with interest in the quality of PMP certification remain satisfied that their needs are served well in the program. ■

Reader Service Number 043