Assessing the quality of project planning
Carrying out a project according to its plan does not necessarily ensure a successful outcome. If the planning is faulty, the project will not result in the expected outcome and vice versa; high-quality planning increases the chances that the project will be properly executed and successfully completed. Responsibility for planning lies entirely with the project manager, who must ensure that it is carried out properly to the complete satisfaction of all relevant stakeholders. Therefore, he or she should make sure that not only executions are carried out according to the plan base line, but also that the plan base line is a reliable one.
Planning is the first phase of any project and researches have identified it as one of the critical success factor in a project (i.e., Standish Group Chaos report, 2001). Although its importance is recognized, no focused tool has yet been developed for measuring the quality of project planning.
Since no such models are available, it is beneficial to identify models that are used in similar environments. A family of models, which evaluate the overall ability of organizational processes and has some similarities with project planning, is the maturity models. Those models describe a framework to evaluate the maturity level of an organization (Paulk, 1995). The first maturity model (Crosby, 1979), which concentrates mainly on quality, does not treat planning as a significant component that has to be evaluated. Later models, such as SW-CMM (Software Capability Maturity Model) or the SE-CMM (Software Engineering Capability Maturity Model) evaluate planning process among the other 18 key processes areas. That is, they identify one planning process out of 18 processes, which is around 5%. One may compare this low percentage with 21 planning processes, out of 39 project processes, mentioned in A Guide to the Project Management Body of Knowledge (PMBOK® Guide), adding up to 54%. It is important to stress that PMBOK® Guide deals with processes that should be implemented by the project manager and not other project management related processes, which are supported by the organization in general.
The content of the planning phase in these models is far from being similar to the know-how in the PMBOK® Guide. For example, The CMM questionnaire includes the following question: “Does the project follow a written organizational policy for planning a software project?” (Zubrow et al., 1994). This question is not discussed as part of the PMBOK® Guide. However, it illustrates a potential added value to the PMBOK® know-how. Maturity models refer to it under the title of “Project Organizational Support” which is also referred as “Enterprise Project Management.”
Needless to say, successful project results will not be reached only by proper organizational support. The project manager must follow the planning processes, described in the PMBOK® Guide, in order to create an effective plan. On the other hand, some maturity models have followed the PMBOK® processes, but did not treat the organizational support element (i.e., Ibbs & Kwak, 2000). Thus, the above two components should be included for proper project planning: the use of planning processes as defined by the PMBOK® Guide and project management organizational support. A model, which includes these two components, may be used to evaluate the quality of the project planning processes in an organization, point out weak areas that need to be improved, and thereby elevate the quality of planning.
The development of the model for assessing the quality of project planning is based on knowledge areas from the fields of Project Management, Control, Organizational Maturity, and Organizational Support. The model, called Project Management Planning Quality (PMPQ), consists of the two following components: Project know-how processes are defined as those planning processes executed by the project manager and Organizational support processes generate the means that the organization places at the disposal of the project manager in order to support proper project planning, execution, and completion.
Project planning know-how processes were derived from the nine knowledge areas and the 21 planning processes included in the PMBOK® Guide. Furthermore, a major product within each process was identified. Identification of those products was necessary since the evaluation of planning quality greatly depends on them. Exhibit 1 identifies the planning processes and the products within the knowledge area of “Scope.” As can be seen in Exhibit 1, only two processes were left as planning processes. The initiation process was not included since it is performed before the formal start of project planning. Scope verification is part of controlling processes, as well as scope change control.
Every single process has three major ingredients: inputs, tools and techniques, and outputs. Process “Input” and “Tools and Techniques” are use by the project manager to achieve effective outputs. The process's output is the most important component since it is the sole purpose of the process. Therefore, the quality level of a process can be evaluated by just evaluating the outputs of that process.
Exhibit 1. Demonstrating the Planning Products Definition
Exhibit 2. The 16 Planning Products Included in the Model
A major assumption used in this model is that the quality of the output is a function of the number of times, or the frequency, that this output is generated. “Learning Curve” research has proved that there is ongoing improvement of performance as a function of the number of times that the operation is repeated (e.g., Yiming & Hao, 2000; Snead & Harrell, 1994; Griffith, 1996; Watson & Behnke, 1991). More than that, the “Expectancy Theory Model” claims that one will not repeat a process that has no significant added value to one's objectives (Vroom, 1964). Finally, although much is said today about controlling processes rather than outputs (the whole ISO9000 series), some control models suggest “output oriented control” when it comes to operational processes, such as project management (Veliyath et al., 1997).
In light of the above, the evaluation of the quality of the planning processes in this model is based on the frequency of generating the desired outputs. For example, there are two outputs in the “Scope Definition” process: the “WBS” and the “Scope Statement Updates.” The WBS is a new output, which has not been generated as an output of another process, whereas “Scope Statement Updates” output updates an entity that has already been generated by the “Scope Management Plan” process.
Each output may include one or more products. For example, the “WBS” output includes a “WBS Chart” and “WBS Dictionary” products (chapter 5.3.3 in the PMBOK® Guide). The most important product among those two is the “WBS Chart,” since the other one may be considered in this stage as a generic template, which will be completed during other planning processes. Following that methodology, one major product was defined for each of the 21 planning processes included in the PMBOK® Guide.
Exhibit 3. The 17 Organizational Supporting Products Included in the Model
A questionnaire was built to represent the planning products. The following scale was used for evaluating the use intensity of the products:
5—The product is always obtained
4—The product is obtained quite frequently
3—The product is obtained frequently
2—The product is seldom obtained
1—The product is hardly ever obtained
9—The product is irrelevant to the projects I am involved with
0—I do not know whether the product is being obtained
A pilot study was initiated with the purpose of evaluating the necessity of every single product. Project managers and others who are involved in project environment completed the pilot study. It was found that some products are highly correlated to each other. For example, a high correlation was found among all “Risk Management” products (e.g., Risk Identification, Risk Quantification, etc.), which means that they can be represented by one entity. A major reason for this finding may due to the low use of risk management. A similar phenomenon repeated itself in the “Procurement” knowledge area. As a result of the above analysis, correlated planning products were united, and the number of planning products was reduced. Exhibit 2 shows the final list of the 16 planning products included in the model.
As mentioned above, there are two major groups of processes: processes covered by the PMBOK® Guide, which have been already reviewed, and organizational support processes, which lack a well-defined formal body of knowledge. The PMBOK® Guide briefly refers to organizational support knowledge areas and processes under the section “Organizational Influence” (chapter 2.3). It defines four knowledge areas, named “Organizational Systems,” “Organizational Cultures and Styles,” “Organizational Structure,” and “Project Office.” Few products for the above knowledge areas were offered by the PMBOK® Guide. For example, a trivial product for the “Project Office” is “Existence of Project Office.” Since the PMBOK® Guide list of relevant products for the above four knowledge areas is not conclusive, there is a need to identify more products from other sources of knowledge.
As was mentioned before, a possible source for defining organizational support processes lies in the dozens of maturity models that have been developed in the past few years. Mapping these models, more than an hundred project management processes have been identified. Canceling overlapping between models and deleting processes that are not included in the planning phase of the project have reduced the outcome to 13 organizational support processes. One specific product has been identified for each process. The four processes, presented by the PMBOK® Guide, were added to the list as well, to include a total of 17 organizational supporting products. Those products were grouped into the four supporting knowledge areas, defined by the PMBOK® Guide. Exhibit 3 introduces the four supporting areas and their supporting products.
A major purpose of every model is to evaluate intensities of scenarios, where in this model the objective is to evaluate the intensity of project planning quality. In order to achieve this purpose, relative importance, or weight, has to be assigned to each of the attributes used for the evaluation. Since there is no prior information concerning the relative importance of the different attributes, it is common to use the assumption that all are of the same impact. Applying this assumption to our model, we assume equal weight for the two groups, namely, “Project Plan” and “Organizational Support.” Using the same logic, products within each group were assigned equal weight as well.
All together, there are 33 products in the PMPQ model, 16 relating to project know-how processes and the other 17 to organizational support processes, as described in Exhibit 3. A 33-item questionnaire, which is based on the model, was designed.
The model's reliability was calculated using a number of statistical tests, such as Cronbach alpha and the split-test methods. Results were considerably higher than the minimum value required by the statistical literature. The results were found to be independent of the person answering the questions, i.e., whether it was a project manager or a senior manager.
Exhibit 4. The PMPQ Model Breakdown Structure
Exhibit 5. Validity Tests for the PMPQ Model
The model's validity was evaluated by comparing the overall project planning quality indicator, which was calculated based on the use intensity of the 33 products, with the projects' success, as estimated by a separate set of questions. It was found out that planning quality was highly correlated with the projects' success, as measured by cost, time, performance envelope, and customer satisfaction. The summary of the analysis is presented in Exhibit 5. It was also found that improving the PMPQ measure in one point reduces the cost overrun by 25%. All results came out statistically significant with p-values under .05.
To conclude the previous sections, we can state that the PMPQ model is reliable and valid and can be used to evaluate the quality of project planning. The following sections report a major finding derived from the use of PMPQ model in different companies and environments.
Using the PMPQ Model
Altogether, 282 project managers and others working in a project environment completed the model's questionnaire. The questionnaire was administered in nine companies, where internal project management workshops were conducted. It was also administered in ten additional project management workshops in which participants came from different organizations.
Exhibit 6 shows the results of the project management know-how processes by nine project knowledge areas, known in the PMBOK® Guide.
By cluster analysis it was possible to identify the following three groups of knowledge areas, which are different in their quality score.
High quality areas include “Integration,” “Scope,” “Time,” and “Human Resources.” The score of this group is around 4.
Exhibit 6. Planning Quality for the PMBOK® Guide Knowledge Areas
Exhibit 7. Planning Quality for Four Organizational Support Areas
Medium quality areas include “Cost,” “Procurement,” and “Quality.” The score of this group is around 3.
Poor quality areas include “Risk” and “Communications.” The score of both is around 2.5.
Let's analyze possible reasons for the poor performance of both “Communications” and “Risk Management.” Although “Communications” is recognized as an essential and critical knowledge area, there is relatively little formal project management knowledge to support it. As a result, project managers do not know how to methodologically deal with the area and mostly use their gut feelings. This is not the case with “Risk Management,” where a relevant body of knowledge is available.
The lack of ability of the project manager to execute a high quality performance in the “Risk Management” processes may derive from lack of friendly tools. Another possible reason may be that owners of the work packages, that is, the functional managers, are the ones that should perform at first “Risk Management” in a matrix environment. Since the functional managers are not necessarily skilled in risk analysis, project managers may find themselves in a frustrated situation, where they do not have the basic needs for applying risk management. A similar analysis is performed for the organizational support processes. Exhibit 7 shows the results for the four organizational support areas.
Via a scatter analysis of the support areas, two groups, which are of different planning quality intensity, were identified. The high intensive group consists of “Organizational Systems” and “Organizational Cultures” and the lower intensive group includes the “Organizational Structure” and the “Project Office” areas. In analyzing organizational support processes, it was found that policy support processes such as “selection of project manager” obtained the highest quality score. However, most of the tactical processes, such as “ongoing project management training,” were poorly executed and obtained low quality scores. The only tactical support that organizations offer with a relatively high quality to their project managers is the purchasing of project management software.
Exhibit 8. Planning Quality by Type of Industry
An important finding to be noted is that the average quality intensity of the two groups of areas, that is, project know-how and organizational support, are of the same average. This finding also points to the equal importance that organizations assign to these two areas.
It was also found that planning quality is impacted by the nature of the industry. Engineering and construction organizations showed the highest planning quality, probably due to the similarity of the projects carried out by these organizations. Production and maintenance organizations plan their projects at the lowest level of quality, perhaps due to the difficulty they face in comprehending the basic difference between managing a project in all its uniqueness and handling their day-to-day operations.
Planning quality level for each industry is presented in Exhibit 8.
The PMPQ model proved to be of high validity and reliability. The overall PMPQ score was found to be highly correlated to project success performance. Its use by project managers and other professionals who work in a project environment helped in identifying “Risk Management” and “Communications” knowledge areas as the Achilles heels of PMBOK® Guide knowledge areas. The use of the PMPQ model as a diagnostic tool for individual companies was found to be of high value. It enables identifying those products and processes that obtain a low PMPQ score. As low score products have a negative effect on overall project performance, a company has the ability to identify those products and initiate corrective actions, which will lead to improved products and project performance.
Crosby, P. B. 1979. Quality is Free. McGraw-Hill, New York.
Griffith, T. L. 1996, March. “Negotiating Successful Technology Imple-mentation—A Motivation Perspective.” Journal of Engineering & Technology Management, 13 (1), pp. 29–53.
Ibbs, C. W., & Kwak, Y. H. 2000, March. “Assessing Project Management Maturity.” Project Management Journal.
Paulk, M. C., Curtis, B., Chrissis, M. B., & Weber, C. V. 1995. The Capability Maturity Model for Software. SEI.
Project Management Institute (PMI) Standards Committee. 2000. A Guide to the Project Management Body of Knowledge (PMBOK® Guide) – 2000 Edition. Newtown Square, PA: Project Management Institute.
Site Standish Group—http://www.standishgroup.com/chaos.html
Snead, K. C., & Harrell, A. M. 1994, July-August. “An Application of Expectancy Theory to Explain a Manager's Intention to Use a Decision Support System.” Decision Sciences, 25 (4), pp. 499–513.
Veliyath, R., Hermanson, H. M., & Hermanson, D. R. 1997, Winter. “Organizational Control Systems: Matching Controls with Organizational Levels.” Review of Business.
Vroom, V.H. 1964. Work and Motivation. New York: John Wiley & Sons.
Watson, W. E. & Behnke R. R. 1991. “Application of Expectancy Theory and User Observations in Identifying Factors which Affect Human Performances on Computer Projects.” Journal of Educational Computing Research, 7 (3), pp. 363–376.
Yiming, C., & Hau, L. 2000. “Toward an Understanding of the Behavioral Intention to Use a Groupware Application.” Proceedings of the 2000 Information Resource Management Association International Conference, Anchorage, AK, USA, pp. 419–422 Idea Group Publishing.
Zubrow, D., Hayes, W., Siegel, J., & Goldenson, D. 1994, June. “Maturity Questionnaire—Special Report.” CMU SEI-94-SR-7. Software Engineering Institute.
Proceedings of the Project Management Institute Annual Seminars & Symposium
October 3–10, 2002 • San Antonio, Texas, USA