Introduction
IT projects have a track record for not performing to expectation. It is almost daily that one reads news reports of yet another failed, delayed, or over-budget IT project. Because IT has such a poor record for success, much research has been directed at determining how these projects should be managed to ensure a successful outcome. Yet IT projects continue to underperform. In this paper we suggest that a contingency approach to IT project management may yield better outcomes.
A successful contingency model should consider many factors when choosing an IT project development methodology. Some researchers focus on clarity of the problem or clarity of the solution as a guide to selecting a methodology (Wysocki, 2006). Other researchers highlight factors like time urgency (Waller, Conte, Gibson, & Carpenter, 2001), budget (Ewusi-Mensah & Przasnyski, 1991), and size of the project (Martin, Pearson, & Furumo, 2007; Stanley, 1988). Martin et al. measured project size based on the management aspects of a project—team size, number of stakeholders, or vendors (Martin et al., 2007). An alternate view is to measure project size based on the technical size of the project—lines of code, number of modules, and similar (Stanley, 1988). As McFarlan suggested, it is also important to consider the amount of organizational change created by the new system (McFarlan, 1981), as well as the system's perceived strategic value (Cash, McFarlan, & Mckenney, 1992).
This study seeks to build a contingency model of IT project management. Our main proposition is that project type should determine the appropriate methodology for a given project in order for the project to be successful.
Developing a Fit Measure
A systems development study should focus on project outcome. However, this is difficult to measure directly. We developed measures of project outcome from Wixom and Watson's work, which investigated data warehousing success factors (Wixom & Watson, 2001). They operationalized project success as organizational implementation success, project implementation success, and technical implementation success. Other researchers measured IT project success along alternative dimensions, including system quality, user satisfaction, and impact on users and organization (DeLone & McLean, 1992).
We propose two general hypotheses to reflect the overall fit between the project characteristics and the methodology in a given system development project:
- Hα: When the fit between project characteristics and the used methodology is high, system success will be high.
- Hß : When the fit between project characteristics and the prescribed methodology is high, system success will be high
Project Types
After surveying the IT project management literature, we established eight independent variables for this study. They can generally be grouped into one of five classes: (1) clarity, (2) quality, (3) time, (4) size, and (5) complexity. These represent inherent characteristics of the project, not external influences such as resource constraints or leadership style.
Wysocki defined clarity of the business domain as the degree to which the IT project's business domain is covered by a “cloud of uncertainty” (Wysocki, 2006). The more this imaginary cloud covers the business domain, the less clear it is. In this study we assess the clarity of the business domain two ways: by showing respondents figures with varying degree of a “cloud” covering a project representation, and by qualitatively asking respondents about the level of uncertainty involved (Pitts & Browne, 2007; Kardasis & Loucopoulous, 2005; Browne & Ramesh, 2002; Stanley, 1988). Two examples of the business domain in relation to an IT project are recognizing that the business needs to understand a particular business process, or is not reaching as many customers as possible.
Clarity of the IT solution is the degree to which the IT solution is apparent at the onset of the project (Drummond, 2005; Beck, Jiang, & Klein, 2006; Cerpa & Verner, 1996). In this study we measure clarity of the IT solution by when the requirements discovery was completed. Completing requirements discovery later corresponds to a less clear IT solution compared to completing this work earlier, which corresponds to a clearer IT solution (Hardgrave, 1995; Hickey & Davis, 2004; Meso & Jain, 2006). To expand on the previous example of a company recognizing a need to understand a business process, the particular IT solution might be to implement a new data mining system or decision support system.
Criticality is an attribute of project quality. It measures the IT project in terms of what the organization stands to lose if the project should fail (Raghunathan & Raghunathan, 1990). In this study we assess the project criticality as being risk either after the project has been completed or while it is being implemented. We qualitatively ask respondents to rate the severity of potential losses if the system or project should fail (Cardinali, 1998; Premkumar & King, 1992).
Perceived strategic value describes the degree to which the IT project changed the business (Subramanian & Nosek, 2001). Unlike magnitude of change, which assesses operational and tactical changes, perceived strategic value assesses strategic changes and strategic importance of the IT project (Cash et al., 1992; Segars & Grover, 1998; Venkatraman & Ramanujam, 1987). Perceived strategic value measures the IT project in terms of what the organization stands to gain if the project is implemented (Silva & Hirschheim, 2007). It is an attribute of project quality.
Urgency means the haste which the project sponsor needs the completed IT project ready (Waller et al., 2001). It measures how much time the development team had available to complete the IT project in terms of overall business needs (Waller, Giambatista, & Zellmer-Bruhn, 1999). Urgency should not be confused with time constraints, which is how much internal time the IT project team has to complete the project.
Project size has been assessed many ways in the literature. The most common approaches include measuring projects in terms of lines of code, number of software modules, and similar quantitative measures (Stanley, 1988; Martin et al., 2007; Aladwani, 2002; McFarlan, 1981; Yetton, Martin, Sharma, & Johnston, 2000). This is the approach we have adopted for measuring scope size.
An alternative measure of project size assesses the size of a project in terms of number of people involved and number of external agents involved (Martin et al., 2007; Stanley, 1988; Barry, Mukhopadhyay, & Slaughter, 2002). We measure this attribute of project size as project management size.
Magnitude of change describes the degree to which the IT project changed day-today business operations, organizational value chains, strategy, and core beliefs (Tushman & Romanelli, 1985; Bergeron, Buteau, & Raymond, 1991; Silva & Hirschheim, 2007). Based on work by Tushman and Romanelli, we measure the impact of the IT project within the organization (Galliers & Sutherland, 1991), with respect to its value chain (Galliers, Swatman, & Swatman, 1995), and with respect to its external environment (Galliers et al., 1995). Magnitude of change is an attribute of project complexity.
IT Success
The IS community has extensively researched IT project success. A successful project has been defined in countless ways. DeLone and McLean conducted an extensive review of IS success research studies to better define what constitutes IS success (DeLone & McLean, 1992). Based on seminal work by Mason and Shannon and by Weaver, they examined IS success along three dimensions: (1) technical success, (2) semantic success, and (3) effectiveness success.
Wixom and Watson empirically tested the taxonomy developed by DeLone and McLean, and found that system success consists of data quality and system quality (Wixom & Watson, 2001). The authors further found that data quality and system quality can be measured along three dimensions of implementation success: (1) organizational implementation success, (2) project implementation success, and (3) technical implementation success (Wixom & Watson, 2001). An IT system is not successful unless it is accepted into the organization and integrated into workflows. The degree of this success is defined as organizational implementation success. Project implementation success is defined as how well the development team meets deadlines, budgets, and functional goals of the system. Lastly, as Wixom and Watson noted, the system is technically unsuccessful if it runs into technical difficulties, such as integrating new technology into an existing technology infrastructure, or if the system is not as flexible or integrated as necessary (Wixom & Watson, 2001).
With basis in the research literature, we adopt Wixom and Watson's framework for measuring IT project success as a three-dimensional implementation success measure.
Figure 1. A Generic Software Development Life Cycle
Project Methodologies
In this study we are interested in the fit between project characteristics and project methodology. We based the development methodologies in this study on Wysocki's taxonomy of development methodologies (Wysocki, 2006). These development methodologies can be viewed on a one-dimensional scale of methodology linearity. In decreasing Degree of Linearity, the development methodologies in this study are (1) waterfall, (2) incremental, (3) iterative, (4) adaptive, and (5) exploratory.
We do not think of the methodologies in this taxonomy as atomic. Rather, we suggest they are categories of several methodologies that share key traits.
Examining the role of stakeholder feedback can assess the degree of methodology linearity. Decreasingly linear methods show increasing reliance on feedback as an integral part of the development process. These methods will also be more willing to go further back in the development process to incorporate that feedback.
In decreasing order of methodology linearity, we outline the distinguishing characteristics of each of these five classes in the following.
Waterfall models are characterized by a one-shot approach. Each stage of the development is fully completed before starting the next stage. The lack of stakeholder feedback is a key differentiating factor of waterfall models (Wysocki, 2006). The project is planned, executed, and launched without asking stakeholders to review or assess (Madsen, Kautz, & Vidgen, 2006; Gasson, 2006; Wysocki, 2006).
Incremental models are often chosen when the project is too large to implement in a single development cycle (Wysocki, 2006). Incremental models are conceptually identical to waterfall models, but they have two key differences. First, the project team divides the project into smaller implementation projects. Each of these is completed one after another using a waterfall strategy (Wysocki, 2006). Second, the development team asks for user feedback at the end of development. However, the development team is usually only willing to return to the build phase to incorporate this feedback (Madsen et al., 2006; Gasson, 2006; Wysocki, 2006).
Iterative models are hybrids between waterfall models and exploratory models. Iterative models plan, build, test, and implement small units of the project at a time, which is similar to what the project team does if they follow the incremental approach. However, in addition to adding new functionality, iterative models often return to existing units to expand or improve these (Wysocki, 2006). Iterative models employ a check at the end of development to determine if the system has met its goals and is ready for public release. If it is not, the project elicits stakeholder feedback that gets incorporated in the build phase (Gasson, 2006; Wysocki, 2006).
Adaptive models incorporate a good deal of research and exploration while developing the IT solution. Requirements elicitation may take up a substantial part of the project time, and is an ongoing activity because needs are constantly changing (Gasson, 2006; Erickson, Lyytinen, & Siau, 2005). The team collects user feedback at the end of the development cycle, and is often willing to go back to the design phase to redesign the system around the user feedback (Turk, France, & Rumpe, 2005; Erickson et al., 2005).
Exploratory models are pure research and development models. These models focus on searching for the optimal solution to vague or unknown problems. Significant stakeholder involvement and feedback is required (Gasson, 2006; Wysocki, 2006). The willingness of the development team to incorporate user feedback all the way back to the beginning phases of the project distinguishes exploratory models (Wysocki, 2006).
Figure 1 offers a summary of the distinguishing characteristics of each methodology class. The left-pointing arrows indicate how far back in the development cycle the team is willing to go when incorporating feedback.
Research Model
To test the assumption that project characteristics determine the appropriate project methodology we developed the following set of hypotheses:
- H1: Clarity of the business domain will be positively and significantly related to methodology linearity, leading to project success.
- H2: Clarity of the IT solution will be positively and significantly related to methodology linearity, leading to project success.
- H3: Project criticality will be positively and significantly related to methodology linearity, leading to project success.
- H4: Perceived strategic value will be negatively and significantly related to methodology linearity, leading to project success.
- H5: Project urgency will be positively and significantly related to methodology linearity, leading to project success.
- H6: Project management size will be positively and significantly related to methodology linearity, leading to project success.
- H7: Project scope size will be positively and significantly related to methodology linearity, leading to project success.
- H8: Magnitude of change will be negatively and significantly related to methodology linearity, leading to project success.
These hypotheses form the basis for developing a contingency model for IT project methodology. See Figure 2.
Figure 2. Research Model
Towards a Methodology Contingency Model
Research Design
Initial versions of the survey instrument were developed based on the IT project management and software development literature. Business school faculty at California State University, Sacramento and outside certified project managers reviewed the survey to identify problems with the wording, content, format, and procedures. The reviewers returned verbal and written comments about the survey instruments and the survey was modified to accommodate these comments. The multiple phases of instrument development resulted in some reorganization and refinement of the survey and established its face validity.
Pilot Test Design
The resulting survey was pilot tested among eight graduate students enrolled in a seminar in software development project management during the Fall 2007 semester in the College of Business Administration at California State University, Sacramento. Pilot participants were asked to conduct a Q-sort on 20 items related to clarity of the business domain and clarity of the IT solution. These items were selected for a Q-sort to establish construct validity because they were only adapted from the literature in concept. An additional 15 items on degree of formality and willingness to explore were added for similar reasons, and we included five items from other categories to counter selection bias. The Q-sort is an effective and established way to quantify subjective matters (Block, 1961), and is commonly used to establish construct validity for qualitative measures.
Pilot Test Results
Initial results from the pilot test indicated confusion among the study participants about the difference between clarity of the business domain and clarity of the IT solution. There was no confusion regarding items measuring business domain clarity. However, respondents had a strong tendency to see items designed to measure IT solution clarity as representing business domain clarity. When we asked respondents questions about a hypothetical business domain, we often got answers that instead related to possible IT solutions. Additional confusion arose between distinguishing between degree of formality and business domain clarity, and between willingness to explore and IT solution clarity.
The most likely explanation for this confusion is that taking each item out of context made it harder to categorize a given item. The survey clusters all the questions about a construct into the same section, and includes metadata to indicate which section a respondent is working in at any given point. Removing these contextual clues made each item much harder to place. We assume that seeing each item in context will eliminate most of this errant behavior.
Each item scoring a 62.5 % confidence level or lower was revised for wording choices. This included all items where three or more of the eight pilot-study participants miscagegorized that particular item. Of the 40 items included in the Q-sort exercise, 18 fell below this threshold and were revised. Additionally, to increase distinction between the two kinds of clarity in the model, “clarity of the business problem” was relabeled as “clarity of the business domain.” We have scheduled a second Q-sort to test the revised items.
Future Work
To investigate the fit between project characteristics and methodology linearity, 4,773 senior managers from self-described IT companies will be contacted by mail in order to identify systems developers and IT project managers who will agree to participate in the study. These senior managers are chief information officers, functional department heads, and chief developers. The list of these senior managers was obtained from a direct mailing company. We selected the list based on geographic location, and chose organizations from regions across the United States considered to be high-technology hubs. These regions included Silicon Valley and San Francisco Bay Area, California; Seattle/Redmond area, Washington; Raleigh, Durham, Chapel Hill (Research Triangle) area, North Carolina; Northern Virginia; and the Boston, Massachusetts metropolitan area.
The identified contact person will be sent a personal letter asking them to forward the URL to the Web-based survey to as many development project managers within their organization with relevant experience as possible. We chose this approach to increase the potential response rate, as we expect several completed survey responses from each organization this way. The survey is Web-based and takes approximately 20 minutes to complete, and is available at http://survey.spurstream.com/survey.php?sid= 7ACB64&uid=0.
Because we are sampling a large number of organizations across dispersed geographic regions, we do not expect nuisance variables such as age, gender, geographic location, or education levels to influence the results. By extension, we expect that participants will constitute a representative sample of U.S. IT project managers.
Conclusion
In surveying a vast population of software projects, we expect to see that those projects that did a better job at identifying the appropriate Degree of Linearity will enjoy a higher degree of IT project success. We also expect to find that environmental factors (for example, organizational inertia, leadership style, and resource constraints, among others) will negatively influence the choice of Degree of Linearity, leading to IT project failure.
Lastly and most importantly, we expect to find that the category of projects with lowest Degree of Linearity will be the smallest category and consist of the most strategically and financially rewarding projects. If these expectations are validated, our findings will help project managers identify the appropriate methodology to use for a given project. This will in turn help them prioritize projects, as well as determine what will be needed to carry them out. Perhaps the most promising contribution, however, will be that this Software Development Contingency Model may one day provide its users with the assurances needed to venture out more often and with more success into what may prove to be the most rewarding category of projects, those with the lowest Degree of Linearity.