From making sense to making cents
part 3 -- the process
by Joan Knutson, Contributing Editor
LET'S REVISIT THE TOPIC of my January and February PM Network columns, in which I discussed the four plateaus by which to measure project management: (1) Comprehension and Acceptance, which assesses that project management is being used; (2) Application, which judges success by the frequency and the accuracy with which the technical and social components of project management are applied; (3) Influence on the Business, which assesses success by quantifying the business results produced by the existence of the project management discipline; (4) Return on Investment, which appraises success through the use of a standard series of calculations.
Like any good newshound, the astute collector of data about project management capabilties must ask the five W's and an H.
The challenge is to implement a process that will gather, analyze and report the results. In measuring project management success, this article focuses on the process.
To measure project management success in any one of the four plateaus, metrics must be set. These metrics aid in evaluating progress over time, set levels of commitment, and position accountability to reach these targets. Actual metrics are acquired through assessments—individual, 360-de-gree, or organizational; through action plans and performance contracts; and/or through performance monitoring.
The process to monitor, track and manage these metrics attempts to balance feasibility, credibility and simplicity, and consists of four steps: Data Collection; Data Analysis; Results Compilation; Action Plan Implementation. For these four steps to be successful, a discipline must be established which mandates that the project management measurement process be managed and maintained.
Joan Knutson is founder and president of Project Mentors, a San Francisco-based project management training and consulting firm. She can be reached at 415/955-5777. Send comments on this column to email@example.com.
Data Collection. This is critical. Only sound data collected in this step can assure valid reports and credible action plans coming out of the process. It is important to generate a data collection plan before beginning the step. Here are questions that need to be answered in order to create the plan:
Why? Which plateau are we addressing and what information are we trying to ascertain?
What? What type of data do we want to collect— financial, attitudinal, performance of specific competencies, designated project success relative to schedule, resource, cost and/or quality?
How? What medium will we use to collect the data—conduct interviews or focus groups, administer an assessment tool, observe and document the observations?
Where? What is the sphere of collection—the entire project management community or a sampling of the community; internal or external or both?
When? How frequently will data be collected and reported out?
Who? Who will collect and process the information—an internal project management council, a project management center for excellence, or a staff position reporting to senior management?
There is no end to the creative ways in which one can collect data; examples include:
Sending out assessment tools
Picking several work-in-progress projects and conducting an interim audit
Administering a “competency exam” to a select control group of project players
Monitoring each project player's performance and development objectives, which are part of their performance management contract
Establishing personal performance contracts with project players prior to beginning a project or prior to rolling out a project management initiative and then tracking those contracts as the project progresses
Examining various business performance records and operational data
Measuring Project Management Summary
As you evaluate the success or failure of project management in each of the four plateaus using the process described in the article, keep your eyes open for the barriers that inhibit and the enablers that encourage the application of project management. To improve the scorecard, before proceeding to implement a project management measurement system you must remove the barriers and parley the enablers.
The effort costs time and money. Be sure that everyone buys into “the return on the investment” for doing this work.
People may not have the skills to perform project management well. Provide them with the necessary training.
People may not buy-in to the particular project management initiative being measured, because they see no incentives or consequences. “Sell” the initiative as you would “sell” the value of doing any project.
Project management is a discipline that requires discipline. Be sure the organization is ready to adhere to the rigor required.
The focus may be too much on the process of project management and not enough on the people who make project management successful. See that there is a balance between the productivity of the process and the performance of the people.
A project initiative may not have been well scoped out and/or may have been done for the wrong reasons or may have been tried in order to attain the wrong result. If this is true, don't discredit the discipline of project management because of a mistake in judgment.
People are afraid of being evaluated by quantifiable metrics. Show them that this is a benefit, not a consequence.
Other pressures make it difficult to stay the course; in other words, to continue to collect, analyze, compile and react to the data. Be sure that this measurement effort is officially scheduled and funded.
There is a viable process in place to measure the contribution of project management to the organization, thus assuring management that they are making a wise investment. Management gains confidence that project management is an investment and not a draining expense.
Knowing what is working and what is not working allows management to make better priority decisions relative to where to spend their project management maintenance and development dollars.
Knowing what is working and what is not allows management to make adjustments and changes to how project management is being performed and to redesign, re-target or eliminate projects or project initiatives as appropriate.
Everyone involved in the project management community can focus on results, not activity or politics.
Customers and sponsors can now see project management as value-added, not as a time waster.
We, in project management, may be reluctant to implement a measurement system because we think that managers really don't care whether or not project management is successful; managers haven't asked specifically for a measurement system, therefore they are not expecting one; we are all professionals and thus we all do a good job, so there is no need to measure it; project management as a discipline is impossible to measure; or that project management is needed, so why do we have to justify ourselves—we'll continue to get funded no matter what.
In my opinion, this is faulty thinking. More and more money is being spent on project management. Project management is becoming more and more visible. It behooves us to step up and measure what project management is doing for our organizations, to laud those successes, and to continually raise the bar, stretching ourselves to become better and better.
Querying project players, users of the outcomes of projects, senior management, external experts or customers on what percentage of improvement, and/or degree of impact (low, medium, high), and/or increased benefits or reduced expenses are a result of the intervention of project management.
Data Analysis. This step uses the data collected. Here the output performance directly related to project management is evaluated along with how this output performance has generated specific improvements in the production and/or performance within the organization. Approaches to compartmentalizing the data include:
Control Group. Compare the same data elements from one group that is using a specific project initiative as opposed to another group that is not using that initiative.
Trend Analysis. Plot historical data prior to the implementation of project management or of a project management initiative. Then project out what the metrics would be (at predetermined time checkpoints) if a project management intervention were never implemented. At those specified time checkpoints, collect and plot actuals. If actuals are better than projections, intervention is succeeding.
Forecasting. This process is similar to trend analysis, but without historical data to rely on.
The goal is to convert intangible benefits to some tangible interpretation of success. If the intangible benefits are too subjective, don't throw them away. Intangible benefits can be as impressive as tangible ones. Create a list of those intangible benefits and let them speak for themselves.
Results Compilation. Results are compiled by documenting the techniques of the data analysis. Documentation may be in the form of graphic representation, such as line graphs or columnar charts. Tables or matrices also lend themselves to communicating this type of data. Lists that have been sorted and extracted relative to specific segments of data can be meaningful. And straight text, if organized well, can accomplish the purpose.
Action Plan Implementation. This measurement process is not just a one-time snapshot. It is meant to be an ongoing reevaluation and reengineering of the process and an ongoing building of the performance of the people.
Create the action plan by taking the reports compilation and isolating those metrics that are not being met or not being met as quickly as expected. Action plans are created and implemented with frequent review checkpoints until the metric is being accomplished.
When creating these action plans:
All parties involved need to commit to accomplishing their part of the plan
The posture needs to be that the metric is meaningful and that the expectation is to hold people accountable for meeting the metric
If the original metric is found to be unrealistic or unattainable, it should be discarded or changed and
The action plans are the important effort before we recycle back to Step 1 of the process—data collection.
A successful process should be (1) simple, economical and easily implemented; (2) routine, embedded into the daily operational project management process; practical, credible and account for external factors; (3) applicable to all projects; and (4) address both the hard and soft data.
THE PROCESS OF measuring project management success is classic and straightforward: gather the data; analyze it; compile the reports; and implement action plans. We are either measuring the success of a composite of projects or of a specific project management initiative. Once the technique(s) of measurement for each of the plateaus have been chosen, the process above can be implemented and refined as time goes on.
Reader Service Number 012
JULY 1999 PM NETWORK