A pilot study in benchmarking project management
Concerns of Project Managers
Looking into other industries’ project management practices was extremely beneficial, and best-of-best practices were identified that can be put to use in our own environment.
EG&G Idaho, Inc., is a prime operating contractor for the Department of Energy (DOE) at the Idaho National Engineering Laboratory (INEL). The company's 5,200 employees carry out a variety of programmatic and support service functions in nuclear reactor design and development, basic and applied research in physical, chemical, and biological sciences, alternative energy process research and development, materials testing and evaluation, nuclear waste management and operational safety.
Recently the Engineering Department at EG&G Idaho conducted a pilot benchmarking study in project management. As pointed out by John Tuman, Jr. in PMNETwork , project management needs to develop a philosophy and a culture attuned to the needs of the changing corporate environment.
A large percentage of EG&G's project management services are provided by a centralized project management organization within Engineering. Some of the factors that prompted EG&G to become involved in the benchmarking process were its ongoing involvement in TQM, wanting to have an objective measure of its performance in mission-critical support functions, a desire to maintain its competitiveness, to support its strategic planning process, and to have a basis for establishing (legitimizing) improvement goals.
Performance in project management is critical to the success of EG&G's mission. This, supplemented by the perception that improvement could be realized, made project management a likely candidate for a pilot study.
This article is presented in four sections. In the first, a brief overview of the benchmarking process is provided. The second section presents the methods used for implementing each step of the benchmarking process in the project management study. The third section presents a discussion of several lessons learned. The final section presents conclusions.
“Benchmarking is the continuous process of measuring products, services, and practices against the toughest competitors or those companies recognized as industry leaders” (David T. Kearns, chief executive officer, Xerox Corporation) . A benchmarking study maybe conducted to support a number of different initiatives and objectives including performance gauging, new process creation, and process improvement. Another type of distinction made in benchmarking is between differentiation versus cost-based studies. In the former, the primary goal is to establish superiority in a service or product, with cost being a secondary issue. The latter has cost as the primary driver.
There are at least four types of benchmarking that can be conducted: (1) benchmarking against internal operations, (2) benchmarking against external direct product competitors, (3) benchmarking against external functional best operations or industry leaders, and (4) generic process benchmarking . In any case, the benchmarking process includes both the acquisition of process knowledge as well as process metrics (benchmarks). This should not be confused with simply establishing benchmarks. An appropriate golfing analogy is that the benchmark is par; emulating Jack Nicklaus’ stroke represents the benchmarking process.
A few different benchmarking process models exist. However, they all are fundamentally the same. The process model used by EG&G, the Verity model , is shown in Figure 1. The process appears to be serial, while in fact it is a dynamic process whose implementation cart be highly iterative.
All three phases of the process shown in Figure 1 are important, but there are some subtle differences in them that should be highlighted. Typically the planning phase is much less resource intensive than the analysis phase. However, good planning will provide a high amount of added value to the study. The only true measure for success occurs in the action phase with the implementation of process improvements and measurement of their effects.
PROJECT MANAGEMENT APPLICATION
The pilot study was conducted by a team of seven persons from the Engineering Department including three project managers, two managers of project management organizations, and two persons formally trained in the benchmarking process. Referring to Figure 1, Step 1, framing the business issue, was characterized as: Some projects are completed without meeting scope, schedule, cost or customer satisfaction requirements.
In Step 2, the areas of importance (i.e., subfunctions of project management) to be benchmarked were identified and prioritized. This was accomplished by translating the business issue into a problem statement, then using an Ishikawa (cause-and-effect) diagram to focus on the causes (subfunctions) of the problem. Subfunctions examined included training, local control, planning and requirements definition, change control, procedures, and organization structure.
In Step 3, expert opinion, contents of the completed Ishikawa diagram, results from a literature search, and input from other companies were used to establish a set of qualitative and quantitative benchmarking measures. Examples of project management measures are: organizational structure, percent of project manager's time spent performing different project management activities, percent of project cost attributed to project management, performance measures for project success, training program, project management software, and project control methods.
Choosing competitors, Step 4, included a conscious decision to look beyond the walls of DOE, and explore other departments of government and the private sector. A phone screening process was used to gauge each company's performance in project management. Different scenarios for company entry strategy are shown in Figure 2. Best-of-best candidates were identified primarily through using expert knowledge, a literature search, input from a consultant, and the aforementioned screening process. Participants in the pilot study included another DOE contractor, a major DOD contractor, and a major telecommunications company. The proprietary nature of this study precludes identifying participants by name or presenting any specific results (e.g., performance data).
Data acquisition (Step 5) was accomplished through a mailing questionnaire and site visits.
Data analysis (Step 6) is an excellent example of the iterative nature of the benchmarking process, with it transcending every step in the process. For example, performance data may be analyzed and used as a criterion in-the process of framing (selecting) the business.
For this study, Steps 7 and 8 of the process are ongoing.
There were a number of lessons learned from this pilot study. They are categories below as they pertain to the team, the process, and evaluation of the pilot study.
Figure 1. The Benchmarking Process 
(Reprinted with permission of Verity Consulting, Los Angeles, California)
- The benchmarking team needs representation and buy-in up-front, at the level the function resides within the company.
- Having a seven-member team was excessive and made logistics a nightmare. A team of five members or less is recommended.
- Having a blend of functional experts and benchmarking process experts is an efficient team approach.
- Team members’ commitment to the study is a key success factor. The study needs to be a top priority to succeed.
- The team should be educated on the benchmarking process at the outset of the study. Reasons for doing so include ensuring an understanding of the objectives and steps in the process, ensuring adherence to the process, avoiding premature site visits, and ensuring that team members understand the level of commitment they are making.
- A project management tool (e.g., a Gantt chart with individual assignments) should be developed for the study. It provides a means of assigning ownership, establishing buy-in, and maintaining accountability.
- Strategic use of consultants (e.g., initial process training, phone screening techniques, questionnaire preparation, and site visit planning and strategy) is valuable and accelerates the learning curve.
- A trained facilitator should be used in all meetings where an important decision needs to be made.
- There may (was) not be a good set of quantitative measures. However, the process knowledge gained, in itself, is valuable.
- Too many measures is a good checkpoint on whether or not the issue is well focused.
- Only a minimal number of people in key interface points (e.g., phone screening and site visits) should be used. This will help ensure the comparability of qualitative information.
- People skills are extremely important, especially in the phone screening and site visits.
- The efficiency of the phone screening process is maximized by ensuring that the right person(s) in the company is contacted; the contact(s) is strategically located (e.g., the organization where the function being benchmarked resides), clearly understands the benchmarking process, understands the level of commitment required, and is empowered to commit the company.
- The effectiveness of site visits is greatly enhanced by (a) having analyzed all respondent questionnaires before any site visit is made and (b) conducting follow-up phone calls after questionnaires are received.
- Meeting with both function owners and individual contributors should be included on the site visit agenda.
- The cost and duration of the study can vary significantly. Both are dependent upon several factors including the data acquisition methods, commitment of the team members to the study, and the complexity of the function being benchmarked.
- On the basis of this pilot and other studies subsequently initiated, a company's infrastructure for benchmarking needs to be in place early on. This includes having a company-level benchmarking office with ownership of the benchmarking process, training a “core” of benchmarking experts who migrate between benchmarking studies, and establishing a benchmarking network.
Figure 2. Benchmarking Company Entry Strategy
Evaluation of the Pilot Study
- Looking into other industries’ project management practices was extremely beneficial, and best-of-best practices were identified that can be put to use in our own environment.
- The study was not focused enough. Too many subfunctions were examined and the best-of-best proved not to be the best-of-best for all subfunctions.
- The value of site visits was confirmed.
Several valuable lessons have been learned that will greatly enhance the efficiency and effectiveness of future benchmarking initiatives. Having an infrastructure in place is also essential. Some of the keys to the success of a benchmarking project are the team composition and size, having a well-focused business issue, strong planning, up-front empowerment for making changes, and the team's commitment to getting the study completed and improvements implemented.
EG&G Idaho, Inc., is currently conducting other benchmarking studies in strategic business areas using the many lessons learned in the pilot study. Conducting a pilot study, prior to engaging in a more comprehensive benchmarking initiative, was definitely a wise decision. It provided valuable insights into the benchmarking process that can be used to make future studies more efficient and effective.
1. Tuman, John, Jr. 1993. It's Time to Re-Engineer Project Management. PMNETwork, vol. VII, No. 1 (February), pp. 40-41.
2. Camp, Robert C. 1989. Benchmarking-The Search For Industry Best Practices That Lead To Superior Performance. American Society of Quality Control Press, Milwaukee, Wisconsin.
3. The Verity Consulting Group. 1991. A Hands-On Guide to Competitive Benchmarking: The Path to Continuous Quality and Productivity Improvement. Verity Press, Los Angeles, California.
Oren Hester is manager of statistics, reliability, and analysis for EG&G Idaho, Inc., where he has worked the past 13 years in a variety of technical and management positions. He received his B.S. degree in mathematics and M.S. degree in biostatistics from UCLA. He previously worked at the Jet Propulsion Laboratory in the areas of telecommunications and mathematical modelling.