Critical success factor analysis for technology acquisition program management




Richard G. Donnelly, PhD
The George Washington University

Proceedings of the PMI Research Conference
11-14 July 2004 – London, UK


In a previous article (Dobbins & Donnelly, 1998), the authors described the results of a critical success factor (CSF) pilot study, in which certain concerns surfaced. Primarily, these concerns involved inconsistencies in the way that CSFs had been identified in prior research, and the consequent difficulty of employing CSFs in managing programs. Examination of the CSF literature over the past two decades shows that most of the prior studies had the goal of identifying lists of general CSFs, which managers could use “universally.” However, the totality of the literature clearly demonstrates that no usable universal list of CSFs has ever been identified by statistical analysis of survey results, the approach taken in most prior research. Also, many of the CSFs identified in prior research were not stated as activities, the form preferred for defining CSFs because it allows objective measurement of whether they are being addressed properly. Finally, all prior lists deliberately excluded contextually relevant CSFs.

In this paper, we report on results from our ongoing CSF research. We demonstrate that there are significant deficiencies in the identification of CSFs by statistical analysis of survey results, when CSFs are tested in the laboratory of actual program management situations, in this case in program offices involved in acquisition of various technology-intensive systems. We also address a number of fundamental questions about CSFs raised by the observations from our pilot study and the literature findings. The questions addressed herein are: (1) if an activity is critical to success, why is it not mentioned by everyone doing a similar kind of work? (2) can a given manager have one, or more, CSFs that are not relevant to most other similarly tasked program managers? (3) if some CSFs are contextually relevant to only one particular program manager, is it still important for that manager to identify them? (4) should research efforts seek to identify only lists of CSFs generally applicable to all program managers? (5) does a survey of other managers tell a given program manager what must be done for success in his or her own program? (6) if managers rely on lists of CSFs provided by someone else, will they either not discover or tend to ignore other CSFs important to their particular assignment but not included in the list?

In addressing these questions, we achieved a research objective that is altogether different from that of most prior research—we developed a generalized process by which program managers could identify CSFs rather than seeking to identify a superior list of general CSFs. The result was the CSFs process model, an approach by which program managers could identify their own CSFs, determine the constraints applicable to each, and derive a set of measures for each CSF. The generalization is in the process, not in the list. This approach formed the basis for the dissertation (Dobbins, 2000) on which this article is based.

Among the findings reported herein, some of the most significant are these: We found that it is critical to understand the applicable constraints in deriving both the CSF and the manner in which individual CSFs are to be measured. In addition, we found that changes in the constraints signal when changes to the set of CSFs are occurring. Also, we found that the CSFs process model serves as a form of program risk assessment, providing a protocol through which a manager could determine the risk in accomplishing each identified CSF successfully.

As a practical matter, our findings demonstrate that by employing the CSFs process model managers learn how to conceive, plan, and implement their programs in terms of the applicable CSFs. Having learned to think in terms of CSFs, managers can then repeatedly apply the approach to current programs to detect changes in CSFs as the program environment changes, and can apply the approach to subsequent program assignments. Peripheral benefits emerged as well: Senior management can use the process as a training tool to assist promotable junior managers to think like executives. Furthermore, a program organization can enhance the accomplishment of organization-wide goals if multiple levels of management all employ the CSFs process model, since managers are better able to engage in shared and cross-dependent activities and communication in this way.

What Are Critical Success Factors?

In the management science literature, there have been several references to CSFs. The body of existing research indicates that a common framework of understanding does not exist among the various researchers employing the term CSF. To establish a common understanding of the subject matter, and to understand the value of CSFs, one must be familiar with the fundamental definition of CSF. Rockart (1979) first introduced the critical success factor theory and defined CSF:

A) CSFs are “the limited number of areas in which results, if they are satisfactory, will ensure successful competitive performance for the organization. They are the few key areas where things must go right for the business to flourish. If results in these areas are not adequate, the organization’s efforts for the period will be less than desired.”

B) B) CSFs are “areas of activity that should receive constant and careful attention from management.”

It is apparent from this definition that CSFs are high-level management areas of activity that must be successfully accomplished to achieve an organization’s broad goals. They are things critical to overall success but are not a statement of every routine thing that should be done for eventual success. In this sense, it can be seen that a set of CSFs identified for a given large-scale project differ fundamentally from the set of interlinked detailed tasks that must always be completed satisfactorily in the ordinary course of business for successful accomplishment of any project. The distinction can be further drawn by noting that successful accomplishment of a set of CSFs creates an organizational environment conducive to enabling the associated projects to be successfully managed. However, successful management of any project always requires an appropriate project concept, detailed project plan, and effective execution of the plan to achieve time, budget, specification, and customer satisfaction goals.

What Value Is There in Determining CSFs?

The underlying theory of CSF analysis, as originally developed by Bullen and Rockart (1981), lays the foundation for a framework for determining those activities, critical to the success of the organization or project, which must be done well by top management. Armed with this information, the information systems departments can structure a knowledge management system focused on measurement of these critical activities. CSFs can also be used as the basis for strategic planning and for vertical and horizontal organizational communication. Once known, CSFs serve to focus the time of the project manager on those things of paramount importance and help to determine which things should be delegated.

Under the premises of the CSF theory, at any point in time during the project life cycle a manager will typically have 10 or fewer activities that are critical to the success of the project. Some CSFs are intrinsic to the project itself, required by the nature and environment of the project. Others are extrinsic. Some may be outside the project itself but not necessarily related to external agents of influence. These may be activities unique to the manager and driven by his or her own personal needs. For example, a manager who has no experience whatsoever with software development or software acquisition may have a need to learn more about software acquisition management when taking over as the project manager of a software intensive acquisition.

Bullen and Rockart (1981, pp. 16-19) suggested that when interviewing managers to extract from them their view of the world, so that their contextually specific CSF can be identified, the objective is to help the manager make explicit that which is normally only implicit. They state that once the manager knows the organizational goals, it is equally important “to determine, in a conscious explicit manner, what the basic structural variables are that will most affect his success or failure in the pursuit of these goals. These are the critical success factors” (Bullen & Rockart, p. 13).

Many of the more skillful program managers intuitively determine CSFs rather than rely on standard information from their own management information system (MIS) to manage projects. However, where the CSFs are not explicitly identified and recorded, they do not become a part of the project history and are not explicit elements of the management reporting process. Furthermore, the underlying constraints for the CSFs do not command attention, and the CSFs are seldom measured. A successor project manager, given his or her own skill level and background, may be more or less capable of, or interested in, identifying CSFs or may focus on a different set of intuitively perceived CSFs, if indeed any at all. The result is that a given project may encounter wide swings in managerial focus and direction due to the particular skills and backgrounds of the different project managers who will attempt to guide the project to completion, each of them attempting to integrate and manage complex information related to as many as 11 different functional disciplines. In the absence of an active and continuous process of identification of the project CSFs, all management activities and decisions are made without any documented record of those activities critical to project success having been incorporated into the project history.

Once explicitly identified and made part of the project documentation, with the underlying constraints clearly and explicitly stated, the information gathered promotes project stability. Once a set of CSFs has been explicitly identified and communicated, the likelihood that the CSF will be ignored becomes minuscule.

Some Prior Difficulties

Project management success often requires practices that go beyond the ordinary. Attention must often be given to those matters that may be of secondary visibility or even unobvious, but that are nonetheless of critical importance. Management researchers have shown that routine commercial operations practices are often inadequate to guide projects to a successful conclusion. In a study of commercial software development efforts, Girling and McManus (1998) reported that 80% of software systems are delivered late and over budget. Examples included the 1993 London Stock Exchange project, which was terminated after an expenditure of 75 million pounds; the Department of Social Security automation project, which is overrun by 400%; and the $45 million failure of the new MIS for the California Department of Motor Vehicles. To make matters worse, commercial organizations do not seem to learn from their mistakes. The authors report that 60% of the surveyed organizations had terminated more than one project for the same reasons, and 70% of these firms failed to keep any records of failed IT projects. In these firms, reactive management was commonplace.

A similar condition has been reported by Schwartz (1998), who indicates that in the better managed companies, on average, 2% of the software projects are late and 1% of all projects are canceled, and in less effectively managed companies, 85% of projects are late and cancellations can reach 40%. Since CSFs define those activities, managers must do well to achieve project success, a management practice within the organization that is CSF-based will improve the performance of the managers in that organization.

One of the more comprehensive early research studies conducted on CSFs was that done by Boynton and Zmud (1984). Boynton and Zmud conducted two case studies and then reported what they concluded were the strengths and weaknesses of the CSF-based method of supporting MIS planning and requirements analysis. They concluded that CSF analysis has been used successfully to identify key concerns of senior MIS management. They also indicated that, beyond the MIS arena, CSFs can be used in developing strategic plans and identifying critical implementation issues; in helping managers achieve high performance; and, in establishing guidelines for monitoring a corporation’s activities.

The authors also noted that CSF identification had been cited by others, in prior research, as having three principal weaknesses: (1) difficulty in use and therefore not appropriate unless analysts possess the capability to successfully apply the CSF identification process; (2) questionable validity because of potential analyst/manager bias introduced through the interview process; (3) questionable applicability as a requirements analysis methodology because the resulting information model may not accurately represent the deployment environment. A clear description of the “CSF identification process” was not given by the authors.

Having noted the difficulties reported in prior research, an investigation was begun to determine whether there was a way to gain access to the advantages of CSFs while avoiding the problems being consistently reported. In a pilot study (Dobbins & Donnelly, 1998) conducted three years prior to formulating the design for the research discussed herein, and in a subsequent related study (Dobbins, 2002), some of the same problems reported by others surfaced. The objectives of the pilot study were to determine if managers of highly complex defense projects are able to identify the CSFs applicable in their cases and to determine whether the reported CSFs were sufficiently general to apply to all or most projects of the same type. Managers of two groups were surveyed—those managing large embedded systems and those managing automated information systems. The results of the pilot study were that some CSFs were mentioned by multiple managers, but that none of the CSFs were mentioned by all or a majority of managers, even when managers worked in a given domain. This emphasized the difficulty of identifying a set of general CSFs.

The literature on CSF research reveals four key characteristics. (1) All of the past research performed with the objective of discovering general CSFs was done using a survey, as opposed to case methods, consistent with the objective of seeking general CSFs. (2) Past research tended to avoid consideration of CSFs that were primarily contextually relevant under the assumption that such CSFs would fail to be sufficiently general, thereby not serving the broad project management community. (3) The various research studies conducted to discover general CSFs tended to each come up with a different list of entries, even though there were some common or overlapping CSFs identified. As a practical matter, this left an open question as to which entries a project manager should use, and why. (4) Most of the CSFs identified were not stated in the form of activities, contrary to Bullen and Rockart’s (1984) fundamental definition. Little or no attention was given to standards of measurement of the CSFs.

The Rationale Behind the Design of the Current Research

Overall, the process developed for the research described herein was designed to determine CSFs by utilizing the guidance of a skilled and impartial interviewer leading managers through the process steps. This is done by first identifying all elements of importance related to each of 10 process model criteria categories, then grouping all related items of information by topic, and then identifying the CSFs for each topic group in terms of an activity. This enhances the ability of the managers interviewed to grasp the underlying constraints for each CSF, to formulate the measures needed for each, and to communicate the results to others with whom they are directly associated. This eliminated much of the abstractness and other difficulties previously encountered. Those previously encountered difficulties seem to have resulted primarily from the typical error of asking a manager to identify their specific CSFs too early, before they learn how to think in terms of CSFs. This leads them to identify CSFs that are vague, too abstract, not stated as activities, hard for others to grasp conceptually, and which are difficult or impossible to measure.

This difficulty can be seen when general CSFs previously identified are examined. It would be difficult, if not impossible, to determine the constraints underlying CSFs identified as top management support, client consultation, project mission, troubleshooting, well-defined schedule and plan, etc. It is also difficult to see how any of these CSF could be effectively measured. If one were to apply these proposed CSFs to any given project, one would have to identify a whole range of context specific activities needed to implement each, and identify a contextual set of measures for each of the CSFs supporting activities. The effort would be significant.

CSFs are activities, not goals. Sum and Ang (1997) recognized this in a published material requirements planning (MRP) study, in which they stated: “Past survey studies on CSFs in MRP implementation did not examine the contextual elements that make up each factor. This is not surprising because surveys are limited in exploring the contextual issues surrounding the CSFs. Previous studies’ definitions of CSFs were too broad and general to provide useful and meaningful guidelines for MRP implementation. For example, top management support has often been cited as a CSF, but what exactly constitutes “top management support” is not really known. Good performance of the CSFs requires that their elements (or constituents) be known so that top management can formulate appropriate policies and strategies to ensure that the elements are constantly and carefully being managed and monitored. Lack of clear definitions of these CSFs may result in misdirected efforts and resources.”

The issues that had to be addressed were how CSFs could be identified while avoiding the problems noted by Boynton and Zmud (1984), Sum and Ang (1997), and others; whether identification of general CSFs should be the real research objective; and how to make sure the CSFs are stated as activities. Behind it all there was the underlying fact that CSFs must be relevant to a manager if he or she is going to use them, and that means we cannot ignore contextual relevance.

Several conclusions were reached in considering these questions. Merely trying to identify CSFs was not sufficient. We must also know on what each CSF is based in order to test its current and continued validity. CSFs must be formulated as activities so they can be tracked over time and measured. It was also clear that the fundamental premise first stated by Rockart and Bullen (1981), that CSFs must be relevant to the individual manager, was true and the research design had to take this factor into account. This, for all intents and purposes, eliminated survey research.

A case study design was chosen to accomplish two objectives. First, instead of focusing on the identification of CSFs, the research focused on defining a comprehensive general process a manager could use to derive, not simply state, their CSFs. This process would have to guide the manager in the identification of the CSFs, determine the underlying constraints upon which each is based, and determine the measures by which the CSFs could be tracked and evaluated. The design also had to make sure that the CSFs were stated as activities and had to account for the fact that most managers have not been trained to think in terms of CSFs.

To accomplish this, an interview based case study design was chosen that would lead a manager through the different process stages so that in doing so the manager would not only become educated in CSF thinking but would also understand how CSFs can be effectively used. This meant that instead of seeking general CSFs, the objective was to find a general process by which the manager could achieve the design objectives of the study. It also required a determination of whether the application of the process could be replicated with relative ease among various managers of significantly different types of projects. The generalization was in the process, not in the CSFs themselves. If this was successful, general CSFs as well as contextually specific CSFs would emerge from the process application, and neither would have to be deliberately sought. To be successful, the process would have to be applied the same way to each manager interviewed, and the projects selected would have to be very diverse in terms of size, complexity, and domain of application.

The Process Model

To accomplish these objectives, a process-based model was designed and given the name CSF Process Model (CSFPM). Through an interview guided application of the process, shown graphically in Exhibit 1, the constraints underlying each CSF surface as the managers discuss issues of importance relative to each of the 10 criteria categories of the model.

In developing and using the CSFPM to assist managers in identifying their CSFs, a particularly significant difference compared to survey research methods is that the managers interviewed are not asked to specifically identify, or list, their CSFs. Based on our preliminary research, we concluded that, until they have been educated in the CSF process, many managers are not sufficiently familiar with thinking in terms of CSFs to provide a credible list of CSFs for their projects.

The process begins when the manager is interviewed. The first part of the interview gathers general project information. This includes such information as name, position, organization, type of project, project size in dollars or person-months, application domain area, current life-cycle phase, project objective, the user community, whether the project has documented strategic goals, the specific goals, how the goals were determined, and whether a critical path analysis has been done. This information is captured and recorded as Part 1 of the data gathered.

In Part 2 of the interview process, the manager is asked to identify all critical activities related to each of the 10 categories of inquiry in the general criteria model. Each category is defined for the manager and, where appropriate, examples are given. The manager understands that he or she may not necessarily have an activity for each of the 10 categories. They also consider not only what was done or is being done, but what that manager believes should be done or should have been done. The desire is not to constrict the thought process of the manager by the past or current activities actually performed on the program. The manager has to be given the freedom to think in a creative or innovative way if that is beneficial. At the end of this data gathering process, the first interview is terminated.

CSF Identification Process Flow

Exhibit 1. CSF Identification Process Flow

The interviewer then examines all the activities described under each of the 10 categories of inquiry and groups them according to subject or topic. Then the activities in each group are analyzed for internal consistency. If the activities are consistent, they are then evaluated for critical validity. This means that if the interviewer identifies an activity that on its face seems to be irrelevant to success, it is flagged to bring it to the attention of the manager during the follow-on interview to see if it should be modified or discarded. The residual set of activities is then analyzed and a candidate CSF proposed that exemplifies what that full set of common activities is advocating. Each proposed CSF is assigned a number. The grouped set of activities on which each CSF is based is the set of constraints for that particular CSF.

After the initial CSF identification process is completed, the complete set of constraints for all the CSFs is examined for collective consistency. If any critical activity (constraint), required for one CSF, is in conflict with a constraint forming part of the basis for another CSF, then it may not be possible to do both CSFs, and the manager must again examine the activities he or she has deemed critical to determine the root cause factors needed to resolve this conflict. Two activities, both critical to program success, cannot remain in conflict if the program is expected to be successful. If the underlying constraints supporting different CSFs are in conflict, the CSFs are necessarily in conflict. This conflict analysis is an important phase of the analysis process used to determine the criticality and validity of the CSFs.

Following the identification of all activities important in each criteria category, the interviewer examines each set of CSF-related constraints to determine a candidate set of measures and considers how the measurement data should be presented for effectiveness in knowledge communication. Each individual constraint is considered for measurement, and the measures may be either quantitative or qualitative. The collective set of constraint measures related to a given CSF becomes the set of measures by which the success in accomplishing that CSF is determined.

This initial set of information is then provided to the manager in a scheduled follow-up interview. In the meantime, the manager may have been thinking about the interview and may have other information to add. In this second interview, which should be much shorter in duration than the first interview, the manager has the opportunity to examine the results, make any modifications he or she feels are necessary—including how the CSFs are stated, determine whether the candidate set of CSFs are complete, and review the candidate set of measures. During this second interview, the interviewer inquires whether the data necessary to perform each of the stated measures is already available in the program office or, if not, if it will be available at the time it will be needed.

Any changes the manager makes are incorporated and the interviewer then prepares the final report. This final report becomes the manager’s product to use in his or her daily activity and to use as the foundation for a knowledge management process. By determining and recording all three types of information—CSF identification, constraints, and measures—and by making this part of the project office documentation, the managers will be able to incorporate the information into their executive knowledge management system, and use the information to determine when a change to a given CSF is occurring. The key to understanding the need for the change is recognizing when documented constraints supporting a given CSF are no longer valid. The new or changed CSF, and its related constraint information, can then be used as the foundation for revising the knowledge management information system content, the strategic plan, and possibly the organization structure, in any way necessary for the manager to have the best possible information and implementation strategy for managing the project.

These interviews go well beyond the limited process of asking managers to state their CSFs. For many managers who have never been trained in either CSF identification or thinking in terms of CSFs, many of the previously reported problems—identifying CSFs that are vague, not stated as activities, ambiguous, not contextually relevant, obscure, not measurable, and so on, would result. However, one of the advantages of having a skilled interviewer working with a manager during his or her initial application of the CSF Process Model is that the interviewer is not only helping them identify the model products (CSFs, constraints, measures), but is also educating them in the complete CSF analysis process. Each step is explained in detail as the results are provided to the manager.

In addition to leading the manager through a constraint identification process, the interviewer also leads the manager through the second phase of the process, which is to show the manager how the constraints are grouped, how the CSFs are determined, and how the measures are determined. This dimension of constraint identification and analysis leading to measure identification provides the managers with a clear understanding of the total integrated process. This is a natural result of applying a general process for the purpose of determining context specific information.

Dealing with Changing CSFs

CSFs may change over time, not only due to changes in the life-cycle phase, but also due to changing project requirements and other environmental conditions. That CSFs can change over time is not intuitively obvious. One naturally expects that CSFs that are identified for a given project, with their given purpose, will be relatively stable. On some projects this is true. However, conditions to which the program must respond can change, thereby causing a change to the program CSFs. These conditions can be technical, financial, or personnel related. The generalized process leads to the identification of specific CSFs, and this process can be applied at any time for any project. Thus, the successful identification and use of CSFs at any point in the project life becomes primarily a matter of properly applying a general process.

Research Results

Using this model, the interviewer met with 11 managers of highly diverse programs. A comparison of the project cases is shown in Exhibit 2.

Each of the managers, with the aid of the interviewer, was able to determine their CSFs, did so with clarity, stated each CSF in terms of an activity, and was able to define measures for each CSF. None of the CSFs identified are ambiguous or at such a level of abstraction that additional effort is required for the CSFs to be applied to the project. Since each manager interviewed had complete control of the final statement of the CSFs and the measures, there was no interjection of interviewer bias or interpretation in defining the CSFs or the associated measures. In addition, none of the managers interviewed changed any of the CSFs subsequent to the initial definition. Each manager confirmed they understood the process, felt comfortable with the process, and understood the results. Each understood they had the freedom to change any of the candidate CSFs and indicated a willingness to do so if they felt it was necessary. Therefore, none of the previously reported problems found in prior research were exhibited in this research.

Cases Investigated

Exhibit 2: Cases Investigated

There is also interest in the extent to which any given CSF is general, i.e., have particular CSFs been identified within multiple cases by their managers. The CSFs identified by managers within these 11 cases are listed in Exhibit 3.

Analysis of these CSFs shows there are four CSFs that are repeated among the different managers. These four CSFs can be categorized as funds management, maintaining system engineering skills in the program office, maintaining technical competency, and maintaining external agent support. Maintaining technical competency is different from systems engineering skills because technical competency may be very narrowly focused, such as cryptography skills or software analysis skills or skills in aircraft seat technology. Systems engineering encompasses a wider base of skills that requires a knowledge of both hardware and software issues. The four common CSFs identified by the managers of the 11 cases are:

Funds management (Cases 1, 6, 8, 9, 11)

Maintaining system engineering skills in the program office (Cases 1, 3, 10)

Maintaining external agent support (Cases 1, 2, 4, 6, 10, 11)

Maintaining technical competency (Cases 4, 7, 9, 11)

It is no surprise that funds management is a CSF that is found among several managers. The surprise might be that it is not seen for every manager. Although important for all projects, funds management may not be among the paramount issues for some managers because they have already gained significant budgetary support from their superiors or because Congress has fenced the funds for their project.

CSFs Identified by Case Managers

Exhibit 3. CSFs Identified by Case Managers

Maintaining system-engineering support is not an unexpected CSF for highly technical projects. The state-of-the-art technologies involved in some of these projects, combined with the pressures to downsize and reduce budgets, puts a strain on the availability of qualified systems engineers available to the project manager. It is becoming increasingly common to see project managers forced to contract out system engineering support because they are not allowed to bring these skilled workers onto their staff, or because the skilled workers on staff are being lost to industry. System engineering is a special skill that is quite scarce because it requires that someone be skilled in both hardware technologies and software technology issues. It is the system engineers who lead the trade-off studies and lead the development of system requirements documents. It is also unusual to see system engineering support appear as a CSF for managers who are managing programs that are predominantly COTS-based information systems or that are exclusively software development programs.

Maintaining technical competency is important for many programs, and system engineering competency is certainly included in this overall category since it is one type of technical skill needed by some programs. System engineering was separated out from technical competency in this analysis because of the specific need for system engineering skills on particular projects. This need for system engineering skills in the project office has been seen in prior research as well. If system engineering had been included within, and as a subset of, technical skill, we would see the CSFs related to technical skill identified by 7 of the 11 managers, but the real underlying need specifically represented in system engineering, which is not seen by simply identifying it as technical skill, would not have surfaced.

Maintaining external agent support was the most frequently mentioned CSFs, which was not a surprise. It is important to virtually every manager to maintain external agent support because this is often associated with continued funding. External agents may change, and they may be termed stakeholders or similar, but external agent support is important for system acceptance and for funding availability. The external agents considered for this category are the people in positions of authority who have budget or program control either at the service headquarters, the program executive officer office, or the Pentagon budget analysis office. Although users of the system are clearly a type of external agent, CSFs that focused on users were not included in this particular category for analysis purposes because of the difference in the nature and responsibilities of users compared to supporters with authority and money. For example, manager Number 2 has one CSF related to external agents, and one related to users, even though users can be considered as a type of external agent. This kind of differentiation and recognition of roles is important when managers identify their CSFs, because the activity of the manager in response to these two groups is different and is measured differently.

Having examined the CSFs in this way, we can conclude that for Department of Defense project managers certain CSFs will likely be important and common to many of them. We can also see from the breakdown that for any given manager, there will likely be identified some CSFs that are generally applicable to many managers, and some that are highly contextual and apply only to that project or that manager. The only manager who did not have at least one CSF in one of the four general categories noted above was the manager for Case Number 5. Case Number 5 is the deputy project manager for contracting. His task is managing the contracting support for the project. Therefore, that manager’s job is not technology based and would not typically involve one of the four classifications noted above.

Discussion: Use of CSFs by a Management Team

We should not assume that the application of the CSF Process Model is limited to individual managers. Management teams can also gain significant benefit from the use of this process. Being able to use the process at any point in the project is particularly important because in the fundamental suggestions for conducting interviews of managers to assist in their identification of CSFs, Bullen and Rockart (1981) indicated the target level of management is often a CEO or general manager, even though every manager should have a set of CSFs pertinent to their particular level of responsibility.

In a large project, the overall task responsibility, while remaining with the project manager for accountability reasons, is distributed in practice among several lower-level managers. Applying the CSFPM to both project managers and lower-level managers will have significant implications as to how large projects can be effectively managed. When the project manager and other managers on the program all engage in a common process of CSF identification and analysis, the project manager can then significantly enhance the likelihood that all levels of management are using a set of integrated and consistent CSFs as a vehicle for interdisciplinary communication and decision-making, and are each focusing their efforts on those activities most critical for program success.

The process can also be applied to teams of managers coming together in a collaborative way, such as through the use of GroupWise or some other similar environment. By collaboratively working through the process, the management team may surface CSFs that apply to the team as a collective whole, even if not derived by any one individual manager. This is an area ripe for continued research.

Risk Assessment

Once the CSF analysis report is completed, the interviewer is then able to prepare the CSFs risk assessment. The CSFs Assessment of Risk Determination (CARD) model is a database analysis of the same information gathered during the interview process. No additional information is required. This analysis examines each of the identified CSFs in a more quantitative way and in terms of the risk related to successful accomplishment of each CSF. A series of standard questions and issues is used to score each CSF, and each reported score, for each item, is weighted. The result is a summary score for each CSF and for the collective set of CSFs.

The CARD analysis information can be easily evaluated using an ordinary spreadsheet or relational database system. All the information needed to perform a CARD analysis is included in the data gathered from the interviews or from the post-interview analysis, so no additional data gathering is required. The score derived is the product of the value assigned to each of the items of information derived from the interview and the weighting factor assigned to each of those items. Weighting factors are based on importance of the item.

The CARD analysis is divided into two parts. The first part is a summary analysis of the overall project based on five high-level factors. The issues examined are whether for this project:

  • A critical path analysis has been done.
  • Overall strategic goals have been documented.
  • Strategic goals have been determined.
  • Project goals are consistent with the industry trend.
  • Each project life-cycle phase has at least one CSFs identified.

The second part of the CARD analysis is a detailed examination of 15 different aspects of each identified CSF. The collective analysis is then conducted and provided to the manager along with the CSF Process Model report.


Results obtained from application of the CSF Process Model show that it is both general and sufficiently comprehensive for each manager interviewed to be able to determine a set of contextual CSFs applicable to his or her needs. Because the model is general, the associated process by which the CSFs are determined is general, and thus this model-based process can be incorporated into the skill base of the manager. However, the identified CSFs themselves will always be contextually specific and therefore directly useable as they are stated. Some, by their nature, will be generally applicable within a given domain application area. It is not necessary to specifically seek general CSFs. They will be identified as a natural outcome of the application of the model. The general CSFs identified are those applicable to the project.

When managers try to use a given set of CSFs, especially a set provided by someone else, even if they are initially applicable, it encourages the manager not to look vigorously for other CSFs that may also exist and that are not part of the provided list. Given the definition of CSFs, that all are critical to project success, limiting a manager’s consideration to only a predefined set of CSFs is to invite project failure in many of the chaotic environments of management today. Even more importantly, it also creates the impression that it is unimportant for the manager to learn how to think in terms of CSFs, leading to a diminished ability to effectively use CSFs or to recognize an inappropriate CSF. It is best to provide managers with an effective generalized process that allows managers to identify those CSFs that are applicable to their particular projects. This is especially true when the CSFs can be identified at any time, for any level of management, and for any purpose relevant to that management position. The results are usable not only for their primary purpose, but also for the purpose of vertical and horizontal communication among managers across the project.

The standard interview guide assures that the model is applied to each manager the same way. Because of the guidance provided by this process, a manager does not simply list the activities he or she feels are critical, thereby preventing the manager from over reacting to the most pressing current problem or the most recent crisis. Therefore, application of the CSFPM works to prevent the use of CSF identification as a form of crisis management, and leads the manager to view the program from a variety of perspectives which are both tactical and strategic, and that focus on specific issues such as external support, performance, and quality.

Boynton, A. C., & Zmud, R. W. (1984). An assessment of critical success factors. Sloan Management Review, 26 (Summer), 17-27.

Bullen, C. V., & Rockart, J. F. (1981, June). A primer on critical success factors (CISR WP No. 69). MIT Sloan School of Management.

Dobbins, J. H. (2000). On a generalized CSF Process model for critical success factor identification and analysis for technology acquisition program management Unpublished doctoral dissertation, George Washington University.

Dobbins, J. H. (2002). Critical success factor analysis for DoD risk management. Program Manager (May-June), 40-45.

Dobbins, J. H., & Donnelly, R. G. (1998). Summary research report on critical success factors in federal government program management. Acquisition Review Quarterly, 5 (Winter), 61-81.

Girling, R. & McManus, J. (1998). The future of software development in large organizations (A case study in the application of RAD). Management Services (April), 8-16.

Rockart, J. F. (1979) Chief executives define their own data needs, Harvard Business Review, 57:2 (March-April), 81-93.

Schwartz, M. (1998). Project manager priorities. Software Magazine (January 15), 47.

Sum, C.-C., & Ang, J. S. K. (1997). Contextual elements of critical success factors in MRP. Production and Inventory Management Journal (Third Quarter), 77-83.



Related Content

  • Project Management Journal

    Proposing a Multilevel Approach for the Management of Uncertainties in Exploratory Projects member content locked

    By Gomes, Leonardo Augusto de Vasconcelos | Brasil, Vinicius Chagas | de Paula, Rafael Augusto Seixas Reis | Facin, Ana Lúcia Figueiredo | Gomes, Frederico César de Vasconcelos | Salerno, Mário Sérgio Managers of exploratory projects might face uncertainties over long timeframes at different levels (e.g., project, portfolio, organization, and network). Although literature offers some guidance on…

  • PM Network

    Sustainable Change member content open

    By Supekar, Rajeev Japan's energy landscape has turned upside down in the past few years. A country once dependent on nuclear energy was forced to rush into renewables after the 2011 Fukushima Dai-ichi power plant…

  • PM Network

    Ties that Bind member content open

    By Fister Gale, Sarah Health and safety is a goal for all communities. And this year's PMI® Award for Project Excellence winners helped deliver new life in unique ways. One improved training for emergency room doctors.…

  • PM Network

    Legal Aid member content open

    By Ali, Ambreen Big law firms are facing big challenges these days. With the rise of artificial intelligence and data analytics, new low-cost outsourcers are automating routine, high-volume legal work -- and…

  • PM Network

    Cloud Burst member content open

    By Fister Gale, Sarah Change or die. Major telecoms face growing pressure from both consumers and competitors to transform the backbone of their networks. Data consumption is skyrocketing thanks to ubiquitous mobile…