An empirical identification of project management toolsets and a comparison among project types

Abstract

This paper presents the results of an empirical investigation of project management practice. Practice is investigated through the study of the extent of use of a large number of practices, tools, and techniques specific to project management. A sub-sample of 1,296 practitioners participating in a large-scale international survey is used for this paper. The sample size and the diversity of contexts in which the respondents are working render the analysis feasible and the results reliable. The data is analyzed to identify patterns of practice. More specifically, using principal component analysis, the research identifies patterns, which demonstrate that practitioners use project management tools and techniques in groups or toolsets. The empirical statistical identification of these groups can have important consequences for both the development of practice in the field of project management and the study of project management practice. A brief attempt is made to compare results with the A Guide to the Project Management Body of Knowledge (PMBOK^® Guide) knowledge areas and process groups. The paper also shows how practice varies with the management of different types of projects: engineering and construction; business and financial services; Information Technology (IT) and telecommunications; software projects. The identification of these variations has important consequences for practice and for the study of practice.

Introduction

The scope of project management practice includes a large number of practices, tools, and techniques. Dealing with such an array of practices, whether it be for making an inventory, studying, teaching, using or any other purpose, is facilitated by groupings or categorizations. There are many ways to group or categorize project management practices. The PMBOK^® Guide (PMI, 2008) presents practices, tools, and techniques grouped in knowledge areas and process groups. Historically these groupings were developed from discussions among groups of project management practitioners from which a consensus emerged. Turner (2006) has provided justification of the nine PMBOK^® Guide knowledge areas from theoretical premises, but no attempt has been made to empirically validate these groups. According to Hudson and Moussa (2006), the nine knowledge areas of the PMBOK^® Guide are shared by four of the five major project management competency standards around the world, those from Australia, South Africa, Association for Project Management (APM) and Project Management Institute (PMI), though the standards from other countries have more than nine knowledge areas.

Is there a single best way to classify project management tools into knowledge areas? One could always argue that a particular process, practice, or tool should be classified in another or more than one knowledge area, or that a new knowledge area should be created to better represent the reality of practice. As an example, according to the Practice Standard for Earned Value Management (PMI, 2005), the earned value control technique is associated with seven different knowledge areas. In short, different practitioners with different backgrounds from various parts of the world, all using sound and rational arguments could certainly classify tools in knowledge areas differently.

The idea of grouping by knowledge area was developed by teams working on previous versions of the PMBOK^® Guide as a way of classifying elements of project management knowledge for presentation in the document. The original eight knowledge areas seem to have been organized as a function of what is being managed: scope, cost, time, quality, human resources, procurement, communications, and risk. Integration management was added in later editions. These are conceptual groupings. This paper aims to empirically identify a structure that underlies the actual practice of project management by investigating patterns in the use of project management practices, tools, and techniques. Practitioners most likely use their tools in groups. The study of their collective practice suggests that practitioners use toolsets according to some rational, for special functions or specific purposes. One of the objectives of the present research is to empirically explore the existence of these specialized toolsets as they are used by the community of project management practitioners. This exploration also has the objective of rendering the large array of detailed practices more manageable by identifying the groupings that exist in practice. The identification of underlying patterns or structures, such as the one identified in the present paper, can contribute to the development of the field in many ways. Empirically based groupings could facilitate the development of practice, teaching and training, documenting, and researching.

Literature review

This research investigates the practices, tools, and management techniques that are specific to project management. Projects managers may use tools and practices from different management disciplines, but the development of their field justifies research that focuses on the specificity of their practice.

There has been considerable research on project management practice, but very little outside the literature specific to project management. As an example, implementation of proper project management processes has been recognized as a key success factor in innovation management, but which good project management practices lead to success has not been explored more precisely (Miller and Floricel, 2004; Thieme, Song, and Shin, 2003). Within the project management literature, research on practices focuses primarily on small and specific groups of practices. Several studies compare a larger number of practices, but most often in a specific context. Zwikael (2009), Williams (2007), Yang et al. (2006), Zwikael and Globerson, (2006), Winch and Kelsey (2005), McMahon and Lane (2001), Raz and Michael (2001), Zeitoun (1998), Hargrave and Singley (1998), and Thamhain (1998) focused on a specific application area, process group, knowledge area, or a particular aspect of practice. The research on practice does not allow for comparative evaluation of the relative use of the whole body of practices. There have been few studies examining differences in project management practice between industries, project types, and contexts. Those studies tend to examine only a limited range of practices inspired by the seminal work of Pinto and Slevin (1987) and Pinto and Covin (1989). Very few adopt a wider view and attempt to identify general use and usefulness of large numbers of project management practices (Besner and Hobbs, 2008a, 2008b, 2007; Thomas and Mullaly, 2007; Milosevic, 2003; White and Fortune, 2002). The present study adopts such a wider view.

The Data

The analyses and results presented in this paper are part of an on-going research program, the most recent phase of which is financed in part by PMI. The aim of the research program is to develop an empirically based understanding of project management practice and the ways in which practice varies in different contexts. To this end, data has been collected from approximately 2,500 practitioners worldwide. A sub-sample of 1,296 is used for this paper.

More than a hundred practices, tools, and techniques specific to project management have been identified and pre-selected by the authors. This set of tools have been identify from a detailed examination of a number of sources including the PMBOK^® Guide (PMI, 2004), Max Wideman's glossary (2006) and several papers published in Project Management Journal (PMJ) and International Journal of Project Management (IJPM). In contrast with other research, general concepts, and processes (e.g., training programs, performance measurement) have been excluded from this research program. The tools and techniques selected for this research are more specific and closer to day-to-day practice, closer to the things people regularly do. This involves a partial view of project management practice; it restricts the investigation to those well-know tools and techniques that are specific to project management. However, doing so ensures that the practitioners participating in the study easily understand the questionnaire. The tools and techniques are presented in Appendix 1. For each tool or technique the respondents answered the following two questions using a five-point Likert scale:

• How extensively do you use this tool or technique?
• In your opinion, to what extent would more extensive or better use of this tool or technique improve project performance? (referred to as “potential”)

The questionnaires also collected contextual data on respondents (position, education, experience, etc.), on their organizations (geographic region, size, industry, project management maturity, etc.) and on their projects (size, complexity, etc.). These variables allow for segmentation of the data to determine how project management practices vary according to organizational and project contexts. The fact that the sample is split evenly for many of these variables renders the analysis easier and more reliable. This information was also used to identify clusters of practitioners. Data was collected through a Web-based questionnaire with support from the PMI Research Department, several PMI chapters and colleagues in universities around the world. The data was collected in three phases in 2004, 2007, and 2009, respectively. Globally, results were found to be very stable from one phase to the other. The respondents of the present research are mostly between 30 and 50 years old (71.6%). Their current primary role and the average number of years of experience in this role are as follows: team member (9%; 8y); project manager (50%; 8y); program manager/director (31%; 5y); other (12%; 6y). Considering that 85% of the respondents declare experience in at least two of these categories, they appear well qualified to provide valuable information based on their practical experience.

Identification of the Groups of Tools, Techniques and Practices, or “Toolsets”

This research is exploratory and the toolsets identified were not planned to be part of a construct, and there were no pre-conceptions regarding toolsets before proceeding with data analysis. Moreover, toolset identification was not considered at the time the tools were selected for inclusion in the study.

Principal Components Analysis (PCA), a classic data reduction technique typically used when variables are inter-correlated, was chosen as a basic method to support toolset identification. Both orthogonal rotation (varimax) and non-orthogonal rotation (oblimin) PCAs were performed on scores for the extent of use of the 90 practices, tools, or techniques listed in Appendix 1. The sample size of 1071 valid cases is well over Nunnally's (1978) rule of thumb stating that the ratio of sample size to the number of items for exploratory PCA should be at least 10:1. Average use levels vary considerably, from 1.4 to 4.0, based on a scale ranging from 1 “not used” to 5 “very extensive use”.

The number of components to extract was determined using the “eigenvalues greater than one” Kaiser criterion (Fields, 2000). The original PCA produced 14 groups accounting for 63% of the total variance. Factor loading was set to 0.5 but was on some occasions lowered to 0.46 in order to include more items when interpretation of the enriched component appeared clearly comprehensible. Both orthogonal rotation (varimax) and non-orthogonal rotation (oblimin) were tested and the exact same set of factors was found. This result substantiates the robustness of the groupings.

The toolsets identified all show acceptable reliability, with Chronbach alphas ranging between 0.62 and 0.90, which is above the recommended limit of 0.60 for exploratory research in non-mature fields (Hair et al., 1998); the present research is the first of his kind in the project management field. The 14 toolsets identified by the PCA incorporate 60 tools of the 90 tools or two-thirds of the dataset. Therefore, 30 tools were left as orphans (i.e., not incorporated in a toolset by the PCA). Considering the objective described above to reduce the data without losing valuable information, diversity, and completeness of the actual practice, an effort was made to incorporate these orphans in the 14 toolsets already identified or to group them in new toolsets. The procedure is described in the next section.

Toolset enhancement

A panel of 45 experts, all Project Management Professional (PMP)^® certified, was invited to propose groups of tools from the list of 90 tools as follows.

1. Each member of the panel of experts was asked to group the tools according to their own judgment; with no restriction on the number of toolsets or the number of items per toolset.
2. The frequency with which the experts grouped each pair of tools together was computed. Agreement between the experts was defined to be more than 10 experts suggesting the same pair. The result was mapped on a matrix of 90 rows by 90 columns.
3. The pairs of tools identifiable from the PCA results were mapped on a similar matrix. The superimposition of the matrices produced an interesting match of clouds of pairs.
4. All the orphans from the PCA that were matched by expert agreement with a tool composing one of the 14 toolsets were identified. These were treated as suggestions by the experts for enhancing the 14 toolsets identified through the PCA.
5. Each “expert suggestion” was tested by calculating a new Cronback Alpha for the enhanced toolset. If the Alpha increased following the inclusion of the orphan item, this item was incorporated into the toolset, if not, the item was left as an orphan. Six tools (items) were added in this way to one or another of the 14 toolsets.
6. The enhancement of the 14 original toolsets left 24 orphans (30-6). A second PCA procedure was applied on the orphans alone. Researchers’ judgment together with the PCA results lead to the creation of four additional toolsets. One of these new toolsets was combined with one of the 14 original toolsets. This resulted in the identification of a total of 17 toolsets.
7. Four tools were left as orphans and discarded from the research: Work authorization, Project procedures manual, Concurrent engineering, and Fast-tracking/rapid implementation. The last two formed the 15^th toolset of the original PCA, but this toolset was rejected because the Cronback Alpha was only 0.48. Nonetheless this grouping is of some interest in that “Fast tracking/rapid implementation” relies on “Concurrent engineering” to be operationalized. However, “Concurrent engineering” is applied in contexts not related to “Fast tracking”. The suggestions to include them in other existing toolsets were not validated (did not have a positive and sufficient impact on the Alpha); these tools were therefore discarded by the research.
8. The final set of 17 toolsets is complemented by an additional 18^th toolset that could only be identified a posteriori. The reason is that the data related to it were only collected during the first phase of the research. This toolset is associated to cost estimation and has an acceptable Alpha score of 0.72. Since the tools composing this 18^th toolset were removed in the following phases of the study, to maintain the questionnaire at reasonable size, it is not considered in the subsequent contextual analysis of the toolsets.

The 18 toolsets are presented in Table 1. The first 14 toolsets are those identified by the first PCA in decreasing order of their Cronback Alpha scores. The toolsets in bold have been enhanced with items suggested by experts. The next three toolsets (15 to 17) were drawn from the second PCA and expert judgments and number 18 was added after analysis of the data from phase 1. Cronback Alphas and the average level of use for each toolset are shown. As can be seen in Table 1, the toolsets that were added following the PCA of the orphans from the first PCA all have moderate to high scores for average use. They, therefore, represent a significant portion of project management practice.

Table 1: The list of toolsets

The following sections present the composition of each toolset. The levels of use of the items composing a toolset can vary a great deal. The average score of a toolset may not reflect the very high use of a single item that is part of this toolset. In the following sections, the items composing each toolset are ordered by their average level of use. Presenting items according to their level of use allows a better understanding of the extensiveness of practice. As previously suggested, each toolset could be considered as a specialized group of project management practices and tools. The items composing a toolset are used as a set; their levels of use vary together. Since participants (project managers or program directors) manage projects in different contexts, further analysis should reveal the differences in use of each toolset according to specific context and project type. This paper examines one important aspect of this context: the project type. The items in bold are the ones that have been added from expert judgment.

The items of this toolset form a coherent group; all are directly related to risk management. The items composing the present toolset are specific and generally recognized as a subset of project management practices. The identification of the risk management toolset can be seen as a validation of the corresponding knowledge area of the PMBOK^® Guide.

The levels of use of the items in this toolset are relatively close to average and the toolset is median in its level of use. The level of use of each item may in part be explained by the sequence of actions normally related to risk management. Risk identification and documentation is a prerequisite to ranking risks and planning contingency; the latter being the identification of alternative strategies to be used to ensure project success if a specified-risk event occurs. The more complex tasks, such as assigning responsibility for high-ranked risk to a risk owner and representing risk information graphically for analysis, usually come later and show less intensive use than the other practices.

In this research program, different project management software functionalities are considered as different practices or different specialized tools intended for specific needs. Besner and Hobbs (2006) showed that the levels of uses of the different functionalities vary with their complexity and the difficulty of use, as well as with the project context. The present analysis of toolsets confirms these observations and further suggests that two groups of such functionalities can be distinguished: one composed of basic functions and another of more advanced or complex functions (see toolset 3 below).

The decreasing order of items in this toolset clearly shows that the most extensively used functions are the ones that are the easiest to use. Moreover, the use of project management software for task scheduling or for monitoring the schedule do not need much support from the organization; these functionalities are among the easiest ones to use in an independent way by the project managers. The use of project management software to schedule resources and particularly to schedule many projects simultaneously is more complex and needs some organizational support; the levels of use of these items are accordingly much lower. The results of the present research suggest that the implementation and use of these basic functions are strongly correlated. The use of project management software for the most basic needs probably encourages and stimulates the use of the other more complex functionalities, which are part of this group.

The average level of use of this second toolset of software functionalities is the lowest of all the toolsets. Following the discussion above, the items composing this toolset are more difficult to use by a single practitioner since most of them are dependent on the organizational initiative to put in place a system supporting such functionalities. Once an organization has decided to invest in a system able to provide at least one of these functionalities, by investing for example in an “enterprise project management” solution or system, the addition of the other functionalities listed above are much more easily achievable. Scenario analysis is the last item of the list and is probably the one requiring the most time from project managers, scarcity of time may explain the much lower level of use of this tool.

The items of this toolset also form a coherent group, all being directly related to the management of multiple projects at the same time. All organizations are limited in the number of projects that they can simultaneously plan, monitor, and control; limits are often based on resource constraints. Project selection and project portfolio analysis are strategic tools aimed to help make better choices considering the many facets of project management. The decisive human resource constraint is integrated into the selection process through the organizational capacity analysis, a tool for planning, tracking, and reporting the alignment between total project commitments and organizational capacity.

Program directors and other higher-ranked managers are more susceptible to use this set of tools than the project managers and team members. This may explain the relatively low-average level of use of this toolset. Its use should be linked with the authority of the respondents and to participation in the initiation and concept phase.

The items of this group are among the least used tools. The use of these tools necessitates much support from the organization. Once the database system is in place, it is certainly easier to use the infrastructure for different needs, which explains why the levels of use of these tools vary concomitantly.

Databases for cost estimating can be linked to success measurement. The use of databases for cost estimating differentiates between high- and low-performing organizations (Besner & Hobbs, 2008a). Cost is an important success criterion, and estimation of cost is therefore crucial to success; the more the initial cost constraint is established on the basis of past experiences, the more the cost criteria should be a reachable target. Since all databases are related to learning from past projects, they should generally lead to process improvement and greater success rates. Besner and Hobbs (2008a) have shown that these tools are considered to have great perceived potential, i.e., respondents consider that a more extensive use or a better use of these tools would improve project performance.

All the items of this group have a high level of use; the average level of use of this toolset is the highest of all. The initial planning toolset looks like the Swiss Army Knife ™ of planning. It integrates scope, time, communication, and responsibility planning tools, which are used during the initial planning of a project.

The project charter, which provides authority to a project manager to conduct a project, is usually supplemented by the output of the other tools of this set. The project charter was an additional item suggested by the experts in the toolset enhancement process. The outputs from the other tools in this toolset are inputs to the project charter, which also contains inputs from other project management tools and techniques. Many of these other techniques are included in the “business case” toolset discussed below.

This toolset forms a very consistent group centered on contract administration in general and fixed-price contracts in particular. The items of this group have globally a near-average level of use and the same is true for the toolset as a whole. But the items composing this toolset have the highest variance among all tools. This high variance is explained by the very high difference in the level of practice in some contexts compared to others. Besner and Hobbs (2008b) have shown the contrasting level of use of contracts between engineering and construction as compared to IT projects, with significantly higher use in the former. More generally the use of these tools is very high in projects for external customers. The contract management tools composing this toolset are thus used extensively in some contexts and very little in others. The average level of use masks this reality. Tools and techniques related to variable-price contracts form another separate toolset (see toolset 14 below.)

The sponsor of a project is a person who espouses the project and secures for it the necessary support and resources at the top level of the organization. The business case is used to establish business justification for the project and to acquire funding. The business case is therefore a supporting tool for the project sponsor, helping him/her to justify the projects in the best interest of the funding organization. Considering this sponsor role, the experts suggested adding this item to the other five items identified by the PCA.

Needs analysis and project mission statement are also at the heart of the business case. The project mission is a brief summary of the background, purposes, and benefits of the project. It is generally linked to higher-level strategic goals that are well known to top level management. The needs analysis focuses on the clients’ needs during the initiation phase of the project. This analysis contributes directly to opportunity/problem definition and business case development.

Overall the tools and techniques in this set show moderate levels of utilization. Doing a needs analysis, developing the business case at the outset, having it approved along with the project mission statement and funding are important project practices. However, the last item in the list shows less intensive use. It would seem that once the business case is approved it is often not revised at gates.

The most common reference to benefits is related to financial metrics. As can be seen here, financial metrics are used more extensively than non-financial metrics. As the discussion above showed, the development and approval of the business case is a relatively well-established project management practice. However, the measurement of the benefits stated in the business case is a much less well-established practice.

Medium-term post evaluation of success aligns the project goal to the longer-term success of the product rather than the shorter life of project development. Medium-term post evaluations of success focus greater attention and accountability of the project manager and other stakeholders on project benefits, more on effectiveness than on efficiency.

The content of this toolset was derived from the results of two consecutive PCAs, i.e., the initial PCA analysis that used all 90 tools and the second one computed on the 24 orphans. The initial PCA retained only two items for this toolset: the change request and the change control board. The PCA on the orphans suggested the creation of another toolset related to the baseline composed of the five items in bold. The combination of the two was considered coherent and produced an Alpha greater than that of either toolset taken separately. The two are, therefore, presented here as a single toolset.

The resulting baseline change management toolset highlights the fact that change management is dependent on baseline identification; change must indeed be related to some initial approved plan. Change requests, baseline plans, and the board in charge of the changes are the most intensively used items intertwine at the heart of this toolset. Configuration review and recovery schedule can ensue from changes; recovery schedules, and changes being naturally linked together. Managing change is tightly related to managing contingencies or management reserve. The much lower level of use for the recovery schedule could be related to the fact that lessening the time constraint by re-baselining is much more easily done than the identification of a solution to recover the alignment with the initial baseline.

The first two items in this list show moderate levels of use. They may not be used extensively on their own but they are both integral parts of the scheduling functionalities of project management software. The other two items on the list are used very little.

The items of this group are among the least used tools and so is this toolset. Compare to the initial planning toolset described above the tools of the network planning toolset are specifically time related, analytical, and mathematical in nature. The probabilistic duration estimate (PERT Analysis) is present in most project management textbooks but is neither used nor valued by the vast majority of practitioners.

Somewhat surprisingly, the PCA identified financial evaluations of projects using the techniques identified with this toolset as being used independently from the business benefits metrics toolset, which includes financial business benefits metrics. It would seem that the practice of measuring business benefits evokes a reality that is different for that evoked by financial evaluation of projects. It may be that in contexts where costs and revenues can readily be measured that financial evaluations of projects are a common practice and that in context where benefits are less easily measured that business benefits metrics are more common. It may also be that financial evaluation is associated with the analysis of the project prior to its approval in the initiation phase and that the measurement of business benefits is associated with a post-project phase. Only further analysis may show this to be the case, but at the present time this dissociation of financial evaluation and business benefits metrics remains somewhat of an enigma.

The extensiveness of the use of this toolset related to team management is probably not representative of the importance of the human role in project management and its impact on projects. The human side of project management is a critically important dimension, but is not covered well by traditional project management tools. This may be to a large extent because the methods that lead to high-performing teams may not be specific to project management. The self-directed work team and team building events show the highest levels of use for this toolset. The low level of use of the team development plan may be a neglected part of this important team management toolset. This toolset was doubled in number of items following expert advice, raising the Cronback Alpha from 0.60 to 0.75.

The project war room is a traditional project-management-specific concept referring to a place where the team gathers and where vital project information is displayed for all to see. The project Website could be consider a virtual version of the project war room, while communities of practice would be an extension of the team itself and possibly the virtualization of the functional silos typically supporting the team in a matrix environment.

The gain-share contract is the least used tool of all, and the toolset it is part of has the second lowest average use of all toolsets. Still the variable-price contract toolset may be important in some specific context, but most of the time it is not. The level of use of contract penalties is also very low. Both gain-share contracts and contract penalties have very low scores on potential.

The very poor perception of these tools may be linked to occidental cultural values, which can be related to a long history of litigation and mistrust in contractual relations. Their administration requires careful specification of requirements and close monitoring of performance. Fear of litigation may help explain their infrequent use as both require demonstration of performance, which can be contestable in court. Cost-plus contracts are a little more common in project management, but also need close monitoring of performance.

The tools listed above are all used most often at project closure. The client acceptance form is a formal but easy-to-use tool compared to the others; this may explain why it is the most used item. Lesson learned/post-mortem is a tool identified as a super tool by Besner and Hobbs (2006). This tool has a very high score for both present use and potential in almost all contexts. The lessons learned are generally gathered at the end of the project at the same time as the customer satisfaction survey is carried out. Such surveys are not performed for all projects, but they are often required by ISO quality management plans. Lessons learned can also be linked to quality plans as lessons learned should lead to continuous improvement, which is an integral part of quality plans.

Monitoring project progress is a fundamental function of the project management discipline; many specific tools have been developed to fulfill this function. This is a coherent list of tools all used for monitoring project progress. The progress report is by far the most extensively used of the group; this tool is among the most extensively used tools in almost all contexts. Moreover, the progress report may comprise and use most of the other items in this list in its making.

The stage gate process was designed for control purposes; by reviewing and monitoring the project progress at each stage, decision makers can monitor the progress of the project and take appropriate actions. Project scorecard concept was developed more recently and is also used for monitoring and control purposes as is the earned value approach. Critical success toolsets most often refer to the well-known study by Pinto and Slevin (1987), while the trend report is a generic tool for progress forecasting. Ease of use by the practitioners may again partially explain the items’ hierarchy of use, earned value being definitely the most complex, and progress report the simplest to use.

This toolset has the lowest alpha and indeed it could be argued that each analysis composing this toolset could be a toolset by itself. But considering that the objective of this research is to reduce data and also considering that an alpha greater than 0.60 is acceptable for an exploratory factor analysis, it was decided to preserve this set of tools as the project analysis toolset. Each of these analyses is generally carried on during the front end or the initial phase of the project. The requirements analysis is the principal item considering its extent of use.

This toolset was build a posteriori from a separate database (N= 699) limited to the first phase of data collection. The items of this group are very coherent. The first four items are obviously related to project cost estimation. The last item, life cycle cost, adopts a broader point of view; it considers the product maintenance and disposal costs beyond the project costs, integrating all costs to estimate and analyze the value of the project. Once again the ease of use seems to explain the hierarchy of use.

Comparing toolsets with the content of the PMBOK^® Guide

Both the PMBOK^® Guide and this research report on project management practice. The ways in which their contents were derived are very different. The PMBOK^® Guide is based on a process to identify a consensus amongst practitioners as to what represents good practice. The present research results are based on an empirical measure of the extent of use of practices, tools, and techniques as reported by a large number of practitioners through a survey instrument. The practices in the PMBOK^® Guide are grouped by process group and/or by knowledge area. The practices in this research were grouped based on a statistical analysis of patterns of actual practice. In Table 2, an attempt has been made to map the empirically derived toolsets with the knowledge areas and/or process groups of the PMBOK^® Guide. All of the toolsets except one have been mapped to at least one knowledge area or process group and many have been mapped to both. Multi-project management is outside the scope of the PMBOK^® Guide and is covered extensively in other standards.

The PMBOK^® Guide does not attempt to differentiate among practices in terms of their extent of use or importance. The research does. In Table 2, the toolsets are presented in decreasing order of extent of use. A very summary attempt has been made to evaluate the relative importance of the tools from each toolset within the PMBOK^® Guide by estimating the extent of treatment within the document as either “extensive” or “summary”. A more systematic evaluation would be a research project in and of itself. The reader may wish to substitute his/her own evaluations for those in Table 2.

Table 2: A comparison of the toolsets with the content of the PMBOK® Guide

Comparisons between the extent of use as reported from the research and the extent of the treatment of the same tools within the PMBOK^® Guide gives an indication as to its alignment with the dominant patterns actually found in practice. For several toolsets, extensive use is paired with extensive treatment in the PMBOK^® Guide. There also appear to be some mismatches; The PMBOK^® Guide seems to overemphasize some practices and underemphasize others. The toolsets associated with the initiating process group saw moderate to extensive use but receive only summary treatment within the PMBOK^® Guide. At the same time, some toolsets show infrequent use but receive more extensive treatment within the PMBOK^® Guide; these include tools and techniques in the planning process group: advanced project management software use, and network planning. The evaluations are of too summary a nature to support recommendations, but these results should provide some food for thought for those involved in producing and updating project management standards.

The research results show a great deal of variation in the average level of use of toolsets and in the levels of use of tools and techniques composing each toolset. Variations in level of use may at least in part be explained by the following factors as described in the previous sections: 1) the complexity and difficulty of use; 2) the support of the organization; 3) the sequence of actions normally related to the toolset function or purpose, i.e., complex analyses that come later in the management process and that need some investment from the organization show much less extensive use than other practices. Understanding more precisely what practitioners really do should allow better management of the limited resources available. Project management development and training could be advantageously designed from a toolset point of view instead of a knowledge areas standpoint. Since a toolset is based on the coordinated use of specific tools found in practice, greater coherence in competence and capability development should be achievable via toolsets management. As will be shown in the next section, different contexts lead to different configurations of toolsets. Such configurations represent precise patterns of practice adapted to these contexts. The toolsets are tangible patterns of actual practice; they can actively be combined to better fit the dynamic organizational strategy. Furthermore, organizations and their project managers can configure their project management practices as a means to build a strategic asset.

Comparing practice between project types

A sufficient number of respondents in four different project types allows for comparisons between the following sub-groups: business and financial services (15%), engineering and construction (14%), information technology and telecommunication (44%), computer software and data processing project (9%); the last 18% of the sample includes a variety of other project types. Here, project type is distinguished from the industry; the type of project deliverable is considered more representative of the patterns of variation in practices than the industry, since for example a construction project can be carried out by a firm in the IT industry and vice-versa.

The toolsets were introduced as items in a varimax rotation PCA producing orthogonal polynomials in each of which only one toolset is dominant The percentage of variance explained by each toolset polynomial varies from 6.55% to 4.27%. This uniform distribution of explained variance substantiates the toolsets identification and the usefulness of these toolset polynomials in the contextual analysis of project management practice that follows.

Anova tests were used to identify significant differences in the levels of use of toolsets among four types of projects. The results are presented in Table 3, in which the toolsets are listed in decreasing order of extent of use. Two different approaches have been used to examine the differences in use of toolsets among the four types of projects. The results of the first approach are presented in columns 1 to 4 of Table 3. Here a comparison was made between a particular type of project and the rest of the sample. For example, column 1 presents the significant differences in use when comparing business and financial service projects to the rest of the sample. The first row of column 2 shows that the initial planning toolset is used significantly less in engineering and construction projects (0.05 < P < 0.1). A plus or minus sign in columns 1 to 4 indicates significantly more or less extensive use. The second approach is to make pair-wise comparisons among the four types, the results of which are presented in the six right-hand columns of Table 3.

Table 3: Significant Differences in levels of use of toolset across four types of Projects

(*** P < 0.01; ** 0.01 < P < 0.05; *0.05 < P < 0.1)

Table 3 shows 15 of the 18 toolsets. Project ending and project analysis are not shown because no significant differences were found for either. The cost estimation toolset is not included in this analysis because the data was drawn from a separate sample as explained above. The extent of use of two other toolsets varies very little among the four project types, namely: multi-project management and monitoring progress. Care must be exercised in interpreting the results portrayed in Table 3. One must avoid confusion between the identification of the overall level at which a toolset is used and the identification of significant differences in use. Table 3 focuses on the latter. For example, the variable price contract toolset is one of the least used even in the engineering and construction (E&C) project sub-sample despite the fact that it shows significantly greater use.

Comparisons between E&C and IT projects

The most often compared project types are probably E&C projects and IT projects. The discussion will thus start by comparing these two and proceed to other pairs of comparisons afterward. Columns 2 and 3 show the differences in the levels of use in E&C and IT projects compared to the rest of the sample. Contrasting use between E&C and IT projects can be observed for six toolsets; meaning that if a toolset is used more in one project type, this same toolset is used less in the other project type. The two types of projects contrast in their use of the business case, IT using it more often than the rest of the sample and E&C less extensively. In E&C projects, business cases are often established and documented before the project is initiated. The business case also has a tendency to remain stable throughout the project life cycle of E&C projects. In this type of project, the mandate is to meet the pre-specified requirements based on well identified needs. This contrasts with IT projects, in which, analysis to develop the business case is often a long and complex process with several iterations. Identification, elaboration, validation, reevaluation, and re-validation of the needs are important aspects in managing IT projects. These take place over time, as functionality is progressively defined. The initial planning that follows the needs and business case identification is also a contrasting practice, such initial planning is often already done in E&C projects before the project manager gets involved as also suggest the lesser intense participation in the first two phases of the project as mentioned above.

The greater use of both contract-related toolsets shows that contracts provide a powerful coordinating mechanism in E&C projects. IT projects rely more on communicating and organizing. The toolsets of initial planning and team management are used significantly more to ensure coordination. The team management toolset is also used significantly more on IT projects.

The fact that people on IT projects are more familiar and more at ease with IT products may influence their greater use of basic project management software functionalities, but it does not seem to influence their use of databases, as they are contrastingly more present in E&C projects. The contrast about databases may rather be a contrast consistent with the progressive definition of needs and uncertainty about the final product in IT projects opposing the well defined pre-specified product to be built in E&C project. The greater uncertainty is also very consistent with the fact that risk management is another toolset where IT projects show much greater use than in E&C projects.

Comparisons of Business and Financial Services Projects with other types of projects

The Business and Financial (B&F) services projects presented in column 1 are in turn quite different from any of the other project types. Only two toolsets in this case are used significantly more in this project type. As with IT projects, the greater use of the business case toolset indicates that the needs analysis and definition of the project are an integral part of the project. The greater use of the financial evaluation toolset may be linked with the intrinsic nature of this project type.

B&F services projects also distinguish themselves through a lesser use of three toolsets. The lesser use of bids and fixed-price contracts is coherent with the correlation with projects that are internal to the organization. The lesser use of change management may be an indication of a lack in defining of a baseline plan at the outset.

Comparisons of IT-Telecom and Software project types

In the literature, software projects are often associated with IT-Telecom projects as one category. In this study, the size of the sample allows for comparisons between them. These comparisons reveal significant and important differences as can be seen from the right-hand column of Table 3.

IT and telecom projects show significantly greater use of initial planning tools, the business case, business benefits metrics, and financial evaluation. In contrast, software projects use the advanced functionalities of project management software significantly more. The advanced functions being used may be those related to scenario analysis and to the management of the availability of critical human resource skill sets. Software projects seem to be managed more as programs and/or portfolios as is shown by the significantly greater use of the multi-project management toolset.

The greater use of the change management toolset is very coherent with the literature on project software development. The new Agile approaches, used more widely in recent years, are strongly linked to this type of project in which uncertainty and the need for flexibility are a specificity of this type of project. A greater use of change management means that practitioners in this area also deal with a reference baseline even if it is constantly modified over time.

Measuring and comparing practices among different types of projects has revealed patterns of practice. The patterns of practice provide strong support for the idea that different types of projects are managed differently. The results are consistent with what is known about the reality of practice on these types of projects and thus provides face value for the results and demonstrates the usefulness of the approach taken in this research. In addition, the analysis presented here adds a great deal of empirically founded detail in the comparison among project management practices by project type.

Conclusion

This research has had the objective of empirically identifying the toolsets used in the actual practice of project management. Conceptualizing the multidisciplinary field of project management from what practitioners do provides building blocks from which to assemble a practice view of project management.

The research has identified two classes of patterns of practice. First, it has showed that project management practitioners use tools and techniques in groups, which can be identified and studied empirically. The identified toolsets are composed of well-known tools and practices used in day-to-day work by experienced project management practitioners. Identification was made using principal component analysis on a sample of 1,296 responses to a survey questionnaire. Toolsets are used in many different contexts, each with its particular management problems, for which project management practices have been adapted and skill developed in their use; skill that can be documented, learned, and transferred. In this way, organizations and their project managers can configure their project management practices as a means to build a strategic asset.

The second pattern of practice that was identified is the significant variation in practice among different types of projects. The results empirically confirm some well-known assumptions about the differences between project types, but they also sharpen our knowledge of these differences by presenting specific patterns of practice for each type. Comparisons between project types heighten our understanding of the practice within each project type. Because practitioners in similar contexts choose similar combinations of toolsets, it is more than plausible that the choice of toolsets is largely dependent on context. Project type is the only contextual variable that was investigated in the present paper. Further research is needed to investigate how other contextual variables influence variations in the use of toolsets. Within the present research program, multidimensional contextual space will be analyzed using cluster analysis to identity contextualized patterns of practice. The capacity to measure global patterns of practice using toolsets as “practice scales” should sharpen our knowledge and improve our understanding of practice in real complex multidimensional contexts.

The processes for producing standards on practice are largely opinion-based. The reality of practice can be investigated empirically and the empirical results could be used to inform the production and revision of standards. The focus of the present research is to investigate practice. The comparison with the content of the PMBOK^® Guide is not the primary focus and has not been undertaken in a systematic fashion. The comparison did, however, indicate that the research results and the content of the standard have much in common. It also indicated some important differences in the way information is organized and in the emphasis that is put on different elements of practice. These results are not conclusive but are indicative of a possible direction for future research.

Bios

Dr. Claude Besner MBA, PMP

Dr. Claude Besner, PMP, holds a degree in Architecture, a master of business management, and a doctorate in management. He is a tenured professor in the Management and Technology Department at the University of Quebec at Montreal (UQAM). He is a past director of the UQAM Master's Program in Project Management; this program is accredited by PMI's Global Accreditation Center. Dr. Besner has also more than 15 years of experience as consultant and project director. He is active internationally in the project management community and has presented papers at both research and professional conferences organized by project management organizations worldwide. He is responsible for the research on the subjects of processes, practices and new project management tools within the UQAM Project Management Research Chair.

Dr. Brian Hobbs MBA, PMP, Project Management Research Chair

Dr. Hobbs has been a professor at the University of Quebec at Montreal (UQAM) in the Master's Program in Project Management for more than twenty years. This program, of which he is a past director, is accredited by PMI's Global Accreditation Center. He has served terms on both PMI's Standards and Research Members Advisory Groups. He is a reviewer for both the Project Management Journal and the International Journal of Project Management. He has presented many papers at both research and professional conferences worldwide.