EACH PROJECT MANAGER has tools to assess the performance of a project, including timelines, deliverables, budgets, and quality standards. But how can you measure the achievements of the project managers? Specifically, what tool gives you such detail that both those who hire, assign, and promote project managers, and project managers themselves, get meaningful feedback?
Traditionally, project management evaluation is looked at as “on time, on budget, on purpose.” But how competently a project manager completes a project is also critically important. Because each of the large variety of project manager skills can significantly impact the quality of any project, this conventional approach to measuring success just isn’t enough.
The managers of project managers—those ultimately responsible for a project's success—need to know how to assess every aspect of the project manager's performance. And project managers themselves have high stakes in getting a complete and fair review when it comes to knowing their own strengths and weaknesses.
Why Measure Competencies?
Measuring competencies answers the following questions and more:What is the project manager's impact on the people of the organization? Did someone quit because of poor management and direction? Not being respected? Feeling out of the loop? Underused and undervalued? Indeed, measuring a competency such as interpersonal and communication skills could prevent problems that result in people walking away from the project. Certainly, a deficit in a competency such as personal values might antagonize certain groups whose support is needed for future projects. And poor marks in the competency of customer focus could discourage stakeholders from all areas. When competencies are weak, the quality of any project is substantially lowered.
Dick Cochran specializes in the “people side” of project management through his training company, COMStar. He helps managers in high-tech organizations become better leaders through improved people skills and performance measurements. He is a member of PMI's Mile-Hi Chapter, the Colorado Speakers Association, and the National Speakers Association.
Exhibit 1. Raters use set values to assess a project manager's behavior to ensure that all raters are “on the same page.”
Having complete measures for all project managers helps senior managers compare their performances and decide which ones receive assignments in the future. Project managers also realize tangible benefits. Clear evidence of a job well done can lead to more assignments, promotions, and raises.
Using quantitative measurements from a variety of key people, not just the boss, gives a much more complete picture of a project manager's performance. In fact, this feedback is even more critical for a project manager whose team members do not report directly to him. Instead of having the clout of being “the boss,” the manager's power comes from honing communication and leadership skills to a fine point. Measuring competencies in a meaningful way reveals the true value of work.
360-Feedback Tool
In the October 1999 issue of PM Network, Judith Olney wrote, “It's important to measure the ability of all project managers to apply project management skills and capabilities effectively. The only ways to obtain these types of data are through 360-degree assessments or behavior event interviewing.”
The term 360 feedback refers to getting a well-rounded (360-degree) view of what's going on. Input is gathered from a variety of people surrounding the project manager whose skills are being measured. Thus, the best input comes from the boss, peers, team members, direct reports, stakeholders, and others in project support positions that work closely with the project manager.
Customized Competencies
One company I work with found the 360-feedback tool very helpful in measuring the comparative performances of a group of project managers. The manager in charge of the evaluation started by identifying the competencies critical to customer success and then created a customized set with them. This set is specifically related to each project manager's responsibilities in this company. Being able to customize the competencies made the measurement much more relevant than having fixed categories—a strong advantage of the 360-feedback tool.
Your organization might choose different ones than those selected by my client, which were project planning, project management, leadership, people management, business management, interpersonal and communications skills, personal characteristics and values, and client focus.
Behaviors and Measures
Practically speaking, competencies themselves are too broad for accurate assessment. It's difficult to answer a vague question like, “How are the interpersonal and communication skills?” So, for each competency, my client identified several behaviors, which include specific activities that define the competency. For example, the interpersonal and communications skills competency would have these behaviors: keeps people informed, writes effectively, is a good listener, and develops productive one-on-one relationships.
Those individuals who interacted with the project manager whose performance was being measured rated each of these behaviors. Based on seeing that person in action at work, the raters gave a quantitative score to each behavior on the survey.
To increase the validity of the raters' observations and make sure they all use the same scorecard, the behaviors are further defined by measures, which are agreed-upon and observable indicators of a particular behavior. Under interpersonal and communications skills, for example, the observable behavior of “keeps people informed” was broken into three measures that indicate behavior:seeks information from others, provides timely information, and is respected by others as a source.
Assign Values
Then the raters (peers, direct reports, team members, bosses, stakeholders, suppliers, support people, other managers, and so forth) assigned a value from 1 to 7 or N for each behavior (see Exhibit 1).
The assigned value scale is designed from 1 to 7 or N for many reasons. Using an odd number allows an observed behavior to be rated right in the middle. When they have a greater choice than 1, 2, or 3, raters feel that they can be fairer and differentiate more carefully in their ratings. In addition, having an N choice for “does not apply” or “no response” is important in order to get an honest “I don’t know” response. Giving this as a choice, rather than forcing an uncertain one, also makes the data more valid.
Here's how three groups rated the project manager's behavior under the interpersonal and communications skills competency of Keeps People Informed: score from managers, 6.1;score from team members, 4.9;score from peers, 5.5. The assigned numbers show an average of the values that the three categories of raters gave for one behavior. The overall rating for Keeps People Informed was 5.5.
Best Respondents
The best respondents include those who have the most interaction with the project manager—supervisors, team members, direct reports, and so on. In fact, each respondent can provide a different perspective on the project manager's performance. The better the selection of respondents who rate the project manager's activities, the more valuable the feedback. This is where the value of the 360-feedback tool really shines. From here, you can build a complete picture of the performance based on each of the agreed-upon competencies and behaviors.
Confidential Input and Reporting
Doing a traditional performance evaluation for anyone can be a time-consuming process. That's why companies don’t conduct them frequently and often leave them incomplete.
Exhibit 2. The individual graph shows the project manager's composite score from all raters for each of the competencies. The group graph is the average of each competency for all the project managers in the group.
Exhibit 3. The project manager's score from each rating group for each competency is shown on this graph. Included in this graph is the project manager's own rating of himself, as well as the rating from his manager.
Today's advanced communications and computing power, though, helps smooth the process and improve the quality of feedback. How? By putting a 360-feedback survey on a floppy disk or, even better, using the Internet.
Using this tool, each survey takes a rater or respondent only 15 minutes to complete. Data from all respondents are sent to an automated collection area where they are compiled into a comprehensive 40-page report.
Exhibit 4. The detail in this graph provides an opportunity to “mine” lots of useful information. Behaviors B5-B8 are listed under the competency, interpersonal and communication skills. Specific measures are listed under each behavior to help all the raters have the same understanding of that behavior. The value and range of scores for each rating group is also shown.
This process assures confidentiality that in turn assures more thorough and honest feedback from respondents. It also puts the information in control of a disinterested party outside the company, never in the hands of an individual who could abuse the confidentiality requirement.
The report generated allows people to “mine” the data for useful information that can be viewed from a variety of perspectives. In the example shown in the overview graph, a project manager's performance is compared against the average performance of all project managers in the company (see Exhibit 2). Notice that the project manager is rated higher than the group in the leadership competency, but lower in product management and business management. This feedback suggests important areas in which the project manager can improve during the next review cycle.
Information can also be viewed according to the category of respondents, as shown in Exhibit 3. The top line on this graph shows the project manager's self-score. It looks as if the project manager in Exhibit 3 has a consistently high opinion of himself. The point of concern comes from seeing where the project manager's boss rated him—on the bottom line on the graph. Note that the team rating of the project manager's performance is in the middle of the two. This shows a significant discrepancy, so it should be discussed and resolved, taking into account all points of view.
Understanding the scores by individual behaviors is even more valuable. Here's where the “mining” operation turns up the most gold. In Exhibit 4, the last behavior (B8), “keeps people informed,” shows exactly where the project manager's manager sees a need for improvement. The rating on this behavior of the project manager is 2.0.
As these exhibits show, the 360-feedback report helps understand how well project managers perform, in their own eyes and in the eyes of those with whom they work, measured against the key competencies and behaviors necessary to be successful in the job.
Successful Implementation
Satisfied users have found the key to using this 360-feedback tool well is to take time initially to build people's confidence in the process. That involves communicating its purpose clearly to all participants. Exactly for what will the information be used? Will project managers agree to use the feedback as a way to start improving their “people skills” on the job? Will the organization use it to set salaries and grant future projects? Determining these issues upfront will garner the right kind of support needed for the always-sensitive evaluation process.
I recommend that you look at the first 360 feedback conducted for each project manager as a development tool only. It becomes a benchmark for an ongoing evaluation program. It also can help verify the clarity and selection of the competencies themselves.
This tool can be especially effective in measuring change from one evaluation to the next, so it's important to make sure that project managers understand how to interpret the report. It will help them develop a plan for change that can measure improvements on the following 360-feedback survey.
Costs of Using 360-Feedback Tool
When reviewing costs, consider the time and hassle saved by using this tool for development—especially compared with the cost of delaying a project or wasting resources on serious personnel problems because a project manager doesn’t have adequate skills.
Organizations realize a tremendous gain when project managers implement even small improvements in the skills that they use on a project.
The 360-feedback tool incorporates the survey data from 10 to 15 key respondents and produces a 40-page report on job performance, based on a customized set of job-related competencies. Costs vary, based on the number of project managers surveyed (volume discounts available) and the number of respondents. It's best to budget for an average of $500 per subject. This tool easily pays for itself by identifying development needs and areas for change before they negatively impact any project.
Ongoing Benefits
Having a complete, timely, and objective measurement of performance provides a tremendous advantage when managing personnel and all other variables of a project. Information gleaned from the 360feedback tool can be applied in many ways, including:
Identifying mentors and future project leaders
Using the information in succession planning
Establishing training needs, and measuring the results
Reviewing fair compensation practices
Balancing strengths of individuals with organizational needs
Using competencies for hiring.
The information gathered is also vital for each project manager's personal development. Consider it as a gift for both the manager of project managers and for the project managers themselves.
USE THE 360-FEEDBACK tool to create win-win situations. Over time, everyone will come to trust it as a fair assessment tool, and project managers will clearly want it to count toward their performance ratings, salaries, and promotions.
When considering how easy this 360-feedback tool is to use, realize that it takes only 15 minutes for each respondent to complete, it has built-in confidentiality, and it produces comprehensive reports compiled by impartial outsiders. The result? Organizations get a lot more fairness for a lot less effort. ■
Reader Service Number 065
Reader Service Number 182