Introduction
We developed our Project Management Review Process in response to the request of one of our Senior Vice Presidents. This particular Senior Vice President is also the chair of our Project Management Executive Council (PMEC), a group of senior-level leaders, brought together to coach and support our Council, and Center of Excellence and their annual plans and activities. We meet with this team on a quarterly basis to review our plans and the ongoing status. We seek their opinions and counsel for our ideas. They provide an excellent perspective of the business. We regularly share the expressed needs of our Project Managers with them and they, in turn, share their ideas or needs with us.
The question posed by our chair at a PMEC meeting was simple enough, “Do we have, or if not, can you quickly create, a process that we can use to conduct a review of project management?” He had recently assumed the leadership of a new organization and wanted to determine how well project management was being employed. He had hoped the review would help to identify already existing strengths as well as areas of opportunity for improvement. His expectations were to replicate the strengths across the rest of his organization and develop improvement plans to address the rest.
The PMEC members wholly endorsed his request. They all saw great value in such a process for our company. They believed that their organizations could also benefit from its application. They mentioned that since they would be able to acquire a service internally that they would normally acquire from an external source, dollar savings could be realized. The idea was that much more practical! All agreed that we should have a review process available. The Center of Excellence was planning to do these kinds of reviews anyway, so why not develop a process now in response to a PMEC request.
The response to our first reviews was outstanding! The following words are an excerpt from an article our Senior Vice President provided for our monthly newsletter after the first reviews:
“Project management concepts are very useful tools in achieving operational excellence…especially in the delivery of complex cross-functional outputs with less defects, less rework, and less time. Therefore Project Management is an essential capability that we must continue to develop and improve. However, we need a practical way of putting Project Management “concepts” into widespread practice…assessing how we are doing…and identifying targets for improvement. Here's how we are going about that. [We] asked the PMCOE to develop a methodology review process and trial it with a few [of our] projects. The results proved to be extremely beneficial for [our] Project Owners involved in the trial. That's the good news. The better news is that the learnings gained from those reviews can be leveraged across the rest of [our organization] and other units as well.”
Development
The reasons we created our project management review process are numerous. Of course, we wanted to respond to our PMEC's request. We also wanted to be able to assess the overall level of deployment of project management within our company. With a company as large as ours, with its business unit structure, an individual organization or project perspective is the best approach to achieve this learning. This process would provide us with the opportunity to review an organization's projects. In reviewing these projects, we would be able to easily identify areas of strength and could look to share and replicate them across the units.
Our reviews would also provide opportunities to identify areas requiring fortification. The Center of Excellence and the Council would be able to formulate action plans and incorporate them into their annual program plan. On the individual project level, areas of opportunity would be identified, recommendations presented, and plans developed with the organization's leadership to address improvements.
Another intent of the review process would be to minimize the costs associated with project management reviews. Typically, external resources were called upon to review our projects. These resources added unnecessary costs. Opportunities to carry what was learned forward were lost once the consultant completed the review.
A clear benefit of the review process and its application was the ability to determine our project management process methodology acceptance. Since the review process was to be based on the methodology, and since we had an expectation that our Project Managers were following the process to a greater or lesser extent, we would easily be able to assess its application and/or understanding. Further, in a more general sense, A Guide to the Project Management Body of Knowledge (PMBOK® Guide) (Project Management Institute, 2000) processes knowledge and application could also be assessed. Additionally, since our Excellence Awards project team also used the methodology as the basis of their evaluation criteria for the determination of the recipients of the annual Project Management Excellence Awards, maintaining a consistent perspective was of paramount importance.
Another clear advantage of having a review process that could be utilized internally was the ability to learn more about our projects, processes, and our Project Managers. With respect to our projects, we would learn more about the scope of work involved, how information about the project is communicated, and the overall importance to the business. In the process area, we would learn of process gaps, the effect of “silos,” and where bad handoffs occur. Lastly, with respect to our Project Managers, we would learn more about their education and training within the project management arena, their experience level, certification status, the general level of support they receive, their difficulties and needs.
In order to create our review process, we first decided to focus on our project management process methodology. We decided that we wanted a set of questions, based on the methodology that would be used in a face-to-face interview of the Project Managers. These questions would be arranged by process phase and directly address some of the important process steps.
We applied our personal knowledge and experience as well during the development of the questions. The members of the Center of Excellence were all experienced Project Managers. Collectively, they were able to bring years of experience to bear. They were able to reflect on areas of the process that had presented them with the greatest challenges. For some, they focused on project team experiences that were almost certainly of concern to those people that they would be interviewing. Of course, we continually referred to the PMBOK® Guide areas of knowledge. Our process methodology was the PMBOK® Guide—compliant so that this was relatively easy for us to do.
The other members of our review team were experienced auditors and reviewers. Most are ISO9000:2000 registered, as well as, RAB-certified auditors (Registration Accreditation Board). These members brought previous audit/review experience to our discussions about interview process and the development of our question set. In fact, we patterned and based our initial review process on the process they had applied effectively for a number of years.
We ensured that our approach to the question set creation activity was in keeping with the approach undertaken by our Excellence Awards project team with respect to the annual Project Management Excellence Awards process. This team conducted an examination of physical documentation, examples, and forms in their reviews and assessments. We decided to adopt the same approach. We carefully reviewed the various award criteria and the team's review process to ensure alignment.
Our approach was simple: by process phase, first identify the parts of the process that we called our “hot spots”—areas or deliverables that while of significant importance, typically either do not receive the appropriate level of support or are generally neglected or overlooked. We included the results and findings of other internal audits in the determination of our “hot spots.” Incorporating these into our review process would provide us not only with the opportunity to test the depth and prevalence of the findings, but also once identified, provide an opportunity to implement a solution. This would then be followed by the formulation of openended questions that would provide the opportunity to derive an answer from the interviewee. We also included a short description of the process phase with the questions.
Tools
The final tools of our project management review process included a number of items as stated below.
Profile Question Set
This was included with and preceded the Project Management Review Question Set. This tool was used to obtain information about the people that we were interviewing. The answers to the questions in this section helped us establish a profile of the group. Data associated with project management training and certification was collected, together with how long they have been Project Managers, their job responsibilities and a description of their current job.
Project Owner Question Set
This question set was used with the Project Owners to establish a baseline of understanding of project management, the level of acceptance and support, and to obtain their perspective and status of the project/program. Consisting of ten questions, areas investigated included the project's measures of success, the perceived project management strengths and weaknesses, their specific support activities and others.
Project Management Review Question Set
This was the primary tool used during the review and interview to gather relevant data. It was a phase-by-phase set of questions with six or seven questions per phase with the exception of the Executing and Controlling Phase (our Implementation Phase), which had 15 questions.
Project Management Review Feedback Survey
This was used to obtain feedback from those interviewed as to their reaction to, and experience with, the overall review process. This tool provided the review team with information needed to render improvements to the process. It addressed the general categories of the opening meeting, the interviews, and the exit meeting debriefings, as well as provided an opportunity for other general feedback. The survey focused on communication, professionalism, the demonstration of project management knowledge, and the establishment of the “safe” environment.
Implementation
With a review team established, and our newly created question set and review process in hand, we called upon our Senior Vice President to advise him that we were ready to initiate the review process. We wanted to ensure a top-down approach, especially for these initial reviews. With the assistance of a Vice President, the Senior Vice President selected several of his unit's mission critical projects/programs. Project Owners for each of the projects were identified and subsequently notified of the upcoming reviews. The reviews involved an initial kick-off meeting with the respective Project Owners, a follow-up one-on-one interview with the Owners, the determination of people to be considered in scope for the review, a kick-off meeting with all review participants, and the actual interviews. At the end of each project's review, a high-level preliminary feedback session was held with all participants. Each interview concluded with the distribution of an evaluation form to solicit feedback for the purpose of improving the review process. A final report was prepared and reviewed at a follow-up session with Project Owners and executives.
The review team consisted of four reviewers per project or program. Two reviewers were project management subject matter experts from the PMCOE and two reviewers were certified ISO9000:2000 auditing consultants. The team leader was a member of the PMCOE. The ISO9000:2000 auditing consultants were from the same organization as the PMCOE, which made teaming easy. A one-day session was held by the PMCOE subject matter experts to familiarize the auditing consultants with the fundamentals of project management. Project management concepts were explained by reviewing the Project Management Review Question Set. The auditing consultants were coached on the framework of acceptable answers to the questions in each process phase.
Interviewing the Project Owners
Once the Project Owners were identified, the Vice President arranged a meeting with the Owners and the review team leader to discuss the upcoming review. The rationale for the review was discussed as well as providing an understanding to the Owners that the review was to focus on the application of project management processes, and was not a performance review.
After the Vice President's meeting, the leader of the review team individually contacted each Owner to establish an introductory one-on-one meeting. This one-hour session was done either via phone or in person. The Project Owner Question Set was used to gather a consistent set of data from each Project Owner. Once this one-on-one meeting was completed, the data was compiled, analyzed and shared with the review team. This provided the reviewers with an understanding of the Owner's view of the project scope, its status and other pertinent background information.
During this initial meeting, the Owners were asked to provide a single-point-of-contact for logistical coordination, (e.g., directions to the work-site, access into the work-site, interview scheduling, conference room and conference call set-up, etc.). This coordinator was typically selected from the administrative support staff. The Owners identified team members they wanted to be interviewed, either through direct selection or by conferring with their management teams. Those selected were project managers, or in some cases, functional team contributors. All Owners informed their teams of the upcoming review and their expectations relating to participation. In order to assure an open and honest dialog with the review team, the Owners reaffirmed to their teams that they should not have any fear or trepidation about the review. Each Owner selected approximately 10–12 people for each of the projects to be reviewed.
Review Plan
A schedule was created for each project that contained all elements of the review plan. It contained: the site location, date(s) of the reviews, list of people to be interviewed, reviewer(s) assigned to interview each person, interview time/place, mode of interview (inperson or telephone). The tool served two purposes, keeping the review team on schedule during the actual review process and providing a historical record of all the logistics.
Kick-Off Meeting
The kick-off meeting was always scheduled at the beginning of the full day of on-site interviews. All participants were present: Project Owner, review team, and those to be interviewed. If a person to be interviewed was at a remote location, a conference bridge was available for their participation. The kick-off meeting lasted approximately 30 minutes. The Owner provided an overview of the project. The leader of the review team set the stage for establishing the ground rules, reviewing the agenda, providing an overview of the day's upcoming events, and answering all questions/concerns. At the kick-off meeting, everyone introduced himself or herself and provided a short description of their scope of responsibilities in relation to the project. The review team leader provided an overview of why the review was being done, how the measurement tool or question set was created, and the schedule of the events. The review team leader emphasized the fact that the reviews were not in any way a review of an individual's performance. This reassurance was vital in establishing the kind of environment that would foster open and honest sharing. All attending were asked to reconvene at a designated time at the end of the day for a high-level readout of the findings.
Interview
Each interview was scheduled to be two hours in length. It was either performed face-to-face or, in a limited number of cases for remotely located team members, via the telephone. To allow for the viewing and sharing of various project artifacts, the interviews almost always took place in the project team member's workspace or office. The workspace can be as informative as the question set! In most instances two reviewers partnered to perform the interviews, one to ask questions and the other to record the responses. Prior to the interview, the reasoning for the partnering was discussed to dispel any feelings of intimidation.
Each individual interview began with the questions from the Profile Question Set, followed by the Project Management Review Question Set. The review team strictly adhered to the question sets. As the reviewers moved through the Project Management Review Question Set, they would identify the process phase that was being discussed.
When a question in the tool set explicitly asked if something had been performed or created, the reviewer asked for a copy of the item. The item was then added to the project's artifact library. This artifact library served two purposes: a repository of the project's documentation and a catalog of useful tools for information sharing purposes across the corporation.
If an area of the question set was deemed not applicable within the individual's area of responsibility, it was omitted. Once the question set portion of the interview was completed, the interviewers asked if there was anything the individual wished the review team to carry forward. In most cases, the team member shared something to be communicated to upper management. Each review concluded with the distribution of an evaluation form, the Project Management Review Feedback Survey tool, to solicit feedback on the review and reviewer(s) for the purpose of improving the review process. An agreed upon date was set for when the feedback survey had to be returned to a single point of contact for tabulation.
Exit Meeting Debrief
The review team met during the day for calibration of the day's interviews and events. Midway through the interviews, the reviewers discussed among themselves how the reviews were going and if there were any trends in the data being gathered.
After the last interview was completed the reviewers reconvened for approximately one hour to tabulate their findings. At the end of the day, or conclusion of each project review, a high-level preliminary feedback session was performed with the review participants. A non-prioritized collection of the review team's general reaction to the data obtained or common findings was provided.
Review Team Tabulation
Within one day of completing the review, the review team met to tabulate the raw data gathered. Two types of rating scales were used, the first implemented was a simple plus/minus rating scale, the second was a numerical scoring scale. Midway through the reviews the review team switched from the plus/minus scale to the numerical scoring scale.
The plus/minus scale proved to be too limited. It only provided a minus rating if no evidence was present or a plus rating if there was evidence present. After several review tabulations, the review team realized a numerical score would provide a more definitive rating. The numerical score was based on a rating of one to five. 5 represented “Best in class performance,” 4—“Consistent evidence,” 3—“Some evidence of…,” 2—“Reactive approach,” and 1—“No evidence of….” This more definitive rating scale was a lesson learned for the review team. Utilization of a numerical scale provided a more succinct score, thus providing more succinct data that could be provided to the Project Owner and their team.
The team went through the Project Management Review Question set, question by question when tabulating the raw data. All data gathered for each question was analyzed, discussed, and rated by consensus. The findings were compiled in categories such as client satisfaction, up-front involvement, milestone management, issue management, jeopardy identification and resolution, and risk management. Also included in the compilation were the areas of change management, recognition, formal project management training, project management tools and communication of project status.
Once the team completed the numerical scoring, a rating of red, yellow, or green was applied to each area of the project management phases being assessed for a dashboard chart. For the dashboard chart, the Consumer Reports version of full/half circles in red, yellow, and green were utilized.
The final tabulation performed was on the Project Management Review Feedback Surveys received from participants interviewed. The return rate for these feedback surveys was extremely high.
Final Report
The review team created a final report that consisted of an executive summary, summary of findings in both dashboard format and individually bulleted items, and identified strengths and areas of opportunities for improvement. The team also provided recommendations, as well as items or issues to be brought back to the leadership team. The review participant's training and formal certifications were quantified and feedback metrics on the review team and the process, including a sampling of verbatim comments was shared. All the high-level findings originally shared at the end of day review were included in the final report.
A brief courtesy meeting to review the findings with only the Project Owner was held prior to a formal meeting with the Senior Vice President, Vice President, and the Owner to present and discuss the final report. The findings in the final report were not modified as a result of discussion in the courtesy meeting. The intent of the final report was to assist in the development and implementation of corrective and preventative action plans to address the improvement opportunities. The review team identified recommendations and next steps to assist in the implementation of those corrective actions. The outcome of the final report meetings was the Project Owners and their teams translating the review findings and recommendations into workable improvement plans.
Summary
We have proven to ourselves and our “clients” that our project management review process is an invaluable tool. We have successfully applied the tool to several key projects. Areas of strength and areas providing opportunity for improvement have been identified and documented. The PMCOE has performed the reviews as a part of their consulting and support capabilities, at no expense to the business units, thus minimizing costs.
The reviews have provided the opportunity to ensure awareness and understanding of our project management process methodology. We have been able to expand our knowledge of the business through an understanding of several mission-critical projects and programs. Processes that were either not functioning as intended or not well understood have been identified. We have come to better understand the project managers, and have enjoyed the interaction with them. Their candor and willingness to share has made the experience rewarding. They were quite willing to use us as “messengers” to communicate specific items to their leadership. More importantly, we were pleased to see that their leadership readily welcomed and accepted these items.
The decision to partner with our certified auditors turned out to be another important success factor. Their expertise in professional auditing teamed with our expertise in project management has proven to be a winning combination.
By the reaction of the participants, in addition to our findings, we are sure that we have asked the right questions. We effectively used all the tools that were designed and created. The profile question set gathered invaluable data with respect to the project managers and their project management training/certification status. The question set used to acquire the project owner perspective was also used successfully. By utilizing our methodology as a basis for our primary tool, a logical, sequential flow was established for the interviews. The input from the feedback survey has also helped us to modify our approach.
With respect to the actual implementation, we have found that the top-down support was vital in creating an environment of willing participation and sharing, not one of hiding and distrust of the review team. The kick-off meeting established the kind of rapport with the participants that we were seeking. All were put at ease by being up front and setting expectations of what was to transpire during the reviews. We were able to avoid the stigma of being referred to as the “corporate auditors.”
The exit meeting debrief was an effective way to close out the day's activities. Numerous comments were provided expressing amazement as to how we had so quickly and easily assimilated their information.
With respect to the sharing of our findings, there was never a feeling of defensiveness when presenting the results to the Owners, teams, and executives. No one ever looked to refute the findings. In fact, initiatives have been created to address these findings, and additional, subsequent meetings arranged to review the process with other key leaders.
Our confidence in our project management review process was reaffirmed through the participant feedback survey verbatim comments. Some of the more direct comments included:
“I believe your current review process to be the foundation toward establishing fundamental program management processes and the monitoring of standard practices.”
“I was surprised how your team captured the results so accurately. I felt that the interviewer truly listened and took our comments seriously.”
“The review process was an extremely valuable tool that was well timed, the process gave the whole team time to sit back and reflect on where we are versus where we should be…The review process also provided me with a refresher and fostered new ideas.”
We accepted the request from our Senior Vice President to create a project management review process, and responded with the creation of an invaluable tool. We were pleased with all aspects of the review process: the creation of our tool set, the process, the top-down support received, the enthusiasm of the teams reviewed, and the acceptance of our findings. We are pleased that the organizations have addressed our findings by creating plans and/or teams. For 2002, the PMEC has accepted an objective for each PMEC member to conduct at least one review by year-end. In addition, we have recently been asked to do several more reviews.
