Early risk assessment in IT projects

integrating risk research into project management practice

Conference Paper Risk Management, Information Technology 29 July 2014

Taylor, Hazel A. | Artman, Edward

How to cite this article:

Taylor, H. A. & Artman, E. (2014). Early risk assessment in IT projects: integrating risk research into project management practice. Paper presented at Project Management Institute Research and Education Conference, Phoenix, AZ. Newtown Square, PA: Project Management Institute.

Abstract

There is an abundance of research advice and practitioner guidance on how to manage risk in information technology (IT) projects. It seems that everyone knows what they should do with respect to risk management, and yet the promised payoffs from following risk management procedures do not eventuate: across industries and organizations, IT projects have an unenviable reputation, with projects failing outright or at least failing to live up to expectations. Does this failure result from inadequacies in the risk management guidance that organizations rely on, or is it a failure by organizations and their project managers to effectively implement this guidance? In this paper we discuss one organization’s innovative approach to making IT project risk management realistic, practical, and effective.

The IT project risk management approach presented here is the culmination of over five years of collaborative practice research within the Project Management Center of Excellence (PMCoE) of a large, municipal organization. The aim of the initiative was to improve the organization’s IT project success rate, and the PMCoE worked through three practical action research cycles to develop and refine a new approach to early risk assessment. The approach is well founded in research findings, and the PMCoE has been able to demonstrate its effectiveness through careful monitoring of project performance data.

The PMCoE’s contingency-based risk assessment process provides a visual aid for surfacing overall inherent risk at the early stages of IT projects, thereby enabling the proactive implementation of strategies to establish an appropriate level of oversight and to improve foundational elements of the project when it is still malleable. The PMCoE’s early risk assessment process, and the risk spider chart that is the primary tool in this assessment, involves a review of a small set of project dimensions known to be related to project performance, without requiring probability-impact estimates of individual risks, which often amount to little more than guesswork at the stages leading up to project launch. The dimension-based assessment provides an easily measurable determination of the overall risk level of a project at the very early stages of the project, and enables the PMCoE to make suggestions to improve the odds of success from the start. The risk assessment process and risk spider chart tool have now been used on over 100 projects, and the organization has been able to demonstrate a steady improvement in project metrics since 2006 when the work began.

The PMCoE’s early project risk assessment process is readily customizable for different organizations and project contexts, and provides a model for other organizations striving to engage in effective practices to improve IT project outcomes.

Keywords: IT project risk management; risk spider chart; project oversight; IT project success dimensions

Introduction

There is an abundance of research advice and practitioner guidance on how to manage risk in information technology (IT) projects. It seems that everyone knows what they should do with respect to risk management, and yet the promised payoffs from following risk management procedures do not eventuate: across industries and organizations, IT projects have an unenviable reputation, with projects failing outright or at least failing to live up to expectations (Sauer, Gemino, & Reich, 2007; Standish Group, 2005). Does this failure result from inadequacies in the risk management guidance that organizations rely on, or is it a failure by organizations and their project managers to effectively implement this guidance?

Most firms use some kind of risk assessment for their IT projects, using checklists for determining the risks that might threaten a particular project, and applying an evaluation, usually based on estimates of probability and impact, to rank the highest threats. However, there is considerable evidence that, while these risk assessments have a foundation in research findings and best practice recommendations, they are not being effectively applied in practice (Bannerman, 2008; de Bakker, Boonstra, & Wortmann, 2010). We know much less about why organizations are not applying risk management practices effectively, but it seems this is one point where the best practice and research recommendations break down. In theory, it should be straightforward to identify potential risks to a project and to assess these risks for the probability that they will actually arise and the impact if they do arise. In practice, however, accurate quantification of probability and impact of a risk is particularly difficult in the highly uncertain situations that are common for IT projects, especially in the early stages of the project (McConnell, 2006).

In this paper we discuss one organization’s innovative approach to making IT project risk management realistic, practical, and effective. The IT project risk management approach presented here is the culmination of over seven years of collaborative practice research within the Project Management Center of Excellence (PMCoE) of a large, municipal organization, the City of Seattle, Washington, USA. The aim of the initiative was to improve the city’s IT project success rate, and the PMCoE worked through three practical action research cycles, informed by extensive research findings on IT project risk, to develop and refine a new approach to early risk assessment. The PMCoE’s detailed collection and monitoring of project performance statistics was critical to the success of these initiatives, because it provided objective measures of the effectiveness of the PMCoE’s work. With a focus on continuous improvement, the City of Seattle continues to develop and refine its project management processes, and the work of the action research cycles described here has provided a stable foundation for these ongoing incremental improvements.

First, we present an overview of the City of Seattle and the types of IT projects in which it engages. Then we describe the processes that the city has developed to manage inherent risks across its IT project portfolio and point out how other organizations can readily adapt these processes to match their own organizational requirements. Finally, we discuss the ongoing performance measures used by the PMCoE to track their IT project performance progress.

The City of Seattle and its IT Projects

The City of Seattle comprises about 29 departments and municipal offices, has almost 10,000 full-time employees and an additional 4,000 part-time staff. The city provides services to an estimated 600,000 constituents within its boundaries and to many additional people and companies outside the city who need one of the many licenses, permits, or other services that are not exclusive to Seattle residents.

The City of Seattle’s IT projects cover a wide range of initiatives, ranging from small enhancements of previously implemented software to city-wide initiatives affecting thousands of users; from customized package implementations within and across departments to partnerships with state agencies for Internet portals for citizen access to IT services and benefits projects. Smaller projects have budgets in the US$100,000, whereas major projects may have multi-million dollar budgets.

IT Governance

Seattle has a partially distributed information technology governance structure (Sambamurthy & Zmud, 1999). The Department of Information Technology (DoIT) is charged with managing the city’s IT infrastructure to support communications technologies and services, and with providing IT leadership and governance throughout the organization. Within the DoIT, the PMCoE’s mandate includes oversight and monitoring of expensive and complex projects to ensure their completion within budget, schedule, and scope targets, as well as the promotion of consistent use of standard project management practices, with the aim of supporting the development of capable and experienced IT project managers. Individual departments are responsible and accountable for their own unique business software application projects, but share this accountability with the chief technology officer, and are expected to comply with project management oversight provided by the PMCoE. This structure introduces some tensions as individual departments seek to maintain their autonomy, while the PMCoE seeks to establish and coordinate strategic directions and to develop and implement common standards and policies for IT project management across all departments. The PMCoE has addressed this tension by adopting a consultative approach, seeking to enhance the quality of the city’s projects through collaboration with the departmental project managers, with the goal of developing project management competencies through the implementation of a set of standardized project management processes and methodologies across the organization.

Central to the DoIT’s mission was the establishment of a strategic framework of actions, including the development of a comprehensive inventory of Seattle’s IT projects; a project portfolio process to evaluate proposed projects to ensure that choices regarding application architecture, technology, security, project practices, and funding are in line with the city’s overall strategic IT directions; a project risk profiling review process administered by the PMCoE to assess the inherent risk in proposed projects and determine the appropriate level of central project oversight; and provision of project management consultative services and training. In this paper we focus on the risk profiling and subsequent oversight processes.

Dimensions of Project Success

A central challenge for the PMCoE in realizing its strategic objectives was the need to identify inherently risky projects early in the project lifecycle, typically before enough work has been done on the projects to have confidence in detailed risk assessments. At this early stage in the project life cycle, when detailed technical requirements are not available, traditional risk assessment approaches break down, with probability-impact assessments of individual risks amounting to little more than guesswork (McConnell, 2006). The alternative approach adopted by the City of Seattle is to focus on the overall inherent risk in each project. Instead of asking: “What is the probability and impact of each risk that could threaten this project?” The key question for the organization becomes: “How inherently risky is this project?” Projects that have higher inherent risk — regardless of which specific risk factors might apply — receive closer monitoring and oversight, and early actions can be taken to reduce the overall risk. Large, inherently risky projects can be broken down into smaller ones, and the requirements specification stage of complex projects can be split off into a separate project. Management can take steps to ensure the most experienced project managers are assigned to high-risk projects and that adequate sponsorship and mentoring are provided.

In order to assess the level of inherent risk in a project, a careful analysis of research and best practice literature was used to determine six key dimensions — criticality, uncertainty, complexity, size, project management maturity, and stakeholder involvement — associated with project success. The rationale for each dimension, and the attributes used to measure the dimensions, are shown in Table 1.

Two aspects of the dimensions are critical for the effective application of the risk profiling process. First is that each dimension is clearly linked to project performance, with a greater overall risk of poorer project performance for those projects that measure at the higher end of each dimension. The second key aspect is that the dimensions and their attributes can be assessed at the earliest stages of the project life cycle and are easily and objectively measurable, without the need for probability-impact estimates. For example, it does not matter whether the project manager considers that the budget is generous or tight, or how he or she assesses the probability of a cost over-run; if the project is a large one (in the context of the organization’s size) then it has more inherent risk (Martin, Pearson, & Furumo, 2007; Sauer et al., 2007). A 10% overrun on a small project has a much smaller impact on the organization than a 10% overrun on a large project. Projects with more dimensions assessed at the high end of concern (for example, high complexity or low project manager experience) are likely to have higher inherent risk and require closer oversight, even if it is not clear exactly which of the 50 or so risks on typical checklists will actually apply.

Table 1: Project success dimensions, measurable attributes, and rationale.

Dimension	Rationale	Attributes
Criticality	Projects that are more mission critical for the organization will have higher inherent risk (Shenhar, Dvir, Levy, & Maltz, 2001), and determining the criticality of a project gives insight into how much risk should be tolerated in key areas of the project. High scores on criticality and visibility attributes also signal the need to examine all other dimensions to ensure the optimum strength of project disciplines is applied.	Safety/mission criticality Internal visibility External visibility
Uncertainty	Uncertainty in any area of the project increases risk. (Howell, Windahl, & Seidel, 2010; Sommer & Loch, 2004). While uncertainty might initially appear to be a dimension that is not readily evaluated at the project initiation stage, the City of Seattle has found that two attributes — scope uncertainty and technology uncertainty — can be assessed with a reasonable degree of accuracy. An ill-defined scope introduces ambiguity to the boundaries around which the project is structured, planned, and estimated. When the team is unfamiliar with the technology — even if the technology itself is not new — planning and estimation accuracy declines, and the learning process usually uncovers new challenges.	Scope uncertainty Technology uncertainty
Complexity	Complexity is closely related to uncertainty, in that uncertainties about scope and technology are often related to complexities of the proposed solution (Howell et al., 2010; Shenhar, 2001; Sommer & Loch, 2004). The complexity dimension is broken down into six attributes. Four of these — changes to business processes and rules, extent of customization and configuration, data conversion complexity, and application interface complexity — are assessed on a low-medium-high scale. The fifth attribute is the number of external project, process, or system dependencies, and for the City of Seattle, projects with four or more dependencies are viewed as high on this attribute. The final complexity dimension is the span of impact, measured by counting the number of departments or external agencies involved in the project, with five or more groups being viewed as high.	Changes to business processes and rules Customization/configuration Data conversion complexity Application interface complexity External dependencies Span of impact
Size	Size matters! Larger projects inherently carry greater uncertainty and the larger the project in terms of cost, duration, and size of project team, the greater the risk of poorer project performance (Martin et al., 2007; Sauer et al., 2007). High duration, high budget projects are more difficult to estimate accurately. Longer timeline exposes the project to changes in technology, staff turnover, loss of executive support, and constantly evolving rules, regulations and business processes. Larger teams introduce greater communication and coordination challenges. Small variances in large estimates can add up to significant sums quickly.	Duration Budget Team size
Project management maturity	The project management maturity addresses management competency and methodological issues. Projects that are not supported by experienced managers using mature methodologies are often challenged with issues resulting in costly rework and schedule slippage. (Herbsleb, Zubrow, Goldenson, Hayes, & Paulk, 1997; Jiang, Klein, Hwang, Huang, & Hung, 2004; Subramanian, Jiang, & Klein, 2007). In contrast, organizations where projects are led by experienced project managers with a track record of systematic project methodology application generally exhibit higher success rates (Sauer et al., 2007; Standish Group, 2005).	Project manager experience Project management maturity level
Stakeholder involvement	The stakeholder involvement dimension also comprises two attributes — the match between the executive sponsor’s experience and the project, and the extent to which end-user involvement has been planned for and built in to the project plan. Strong executive sponsorship has long been recognized as a critical project success factor (Wallace & Keil, 2004; Wallace, Keil, & Rai, 2004), and this attribute assesses the sponsor’s commitment and availability and prior experience with similar projects. Similarly, end-user involvement has long been recognized as a critical factor in project success (Schmidt, Lyytinen, Keil, & Cule, 2001).	Executive sponsor experience End-user involvement

The quantification of each attribute will vary depending on the organizational context. For example, the budget attribute can be measured in any organization, but the absolute dollar values associated with a large project budget will differ relative to the size of the organization. Similarly, some organizations might choose to merge the two visibility measures, whereas other organizations may argue that their projects never involve a life/safety critical component and so might use organization mission-critical as the maximum on the criticality scale. Organizations that rely on vendors for much of their project work would likely add a dimension for vendor uncertainty, with values ranging from ‘proven in our organization’ through ‘new to us but strong industry references’ to ‘unproven in the industry.’ With appropriate organization-specific customization, the overall set of dimensions is applicable to IT projects in any organization.

Determining the Level of Project Oversight

The project risk profile assessment is designed to assess the degree of overall inherent risk associated with each project, which, in turn, determines the level of oversight required during the life of the project. As shown in Figure 1, after reviewing the project risk profile, the PMCoE makes a recommendation to the chief technical officer (CTO) of one of four levels of oversight. For projects explicitly designated as no oversight required — typically minor projects with minimal risk — the local department is solely responsible for project outcomes. For all other projects, the PMCoE works with the local department to implement the final oversight decision.

Figure 1: Levels of project oversight.

Dashboard Reporting

All projects other than no oversight required projects are added to and tracked through a digital dashboard, with project managers required to report monthly progress on a set of pre-established metrics, including planned versus actual costs-to-date; planned versus actual schedule-to-date; achievement of projected work packages and milestones; and key project risks and issues. The PMCoE analyzes these dashboard reports and reviews them with the departmental project manager, checking for patterns of variance that could trigger more in-depth assessments or a move to a higher level of oversight. The dashboard reporting ensures regular monitoring of project health and also supports the collection of project metrics for finished projects, including cost at completion, and cost and schedule variance at completion.

Milestone Reviews

Projects considered to have medium levels of inherent risk receive additional oversight of critical milestones. An independent quality assurance consultant is assigned to the project to establish a set of significant milestones in consultation with the PMCoE and the department project manager and sponsor. At each of these checkpoints, the project manager and project sponsor meet with the independent quality assurance consultant to conduct a formal review of project management practices, deliverable progress, schedule and budget performance, project artifacts, project issues, and significant risks. Milestone review findings and any recommendations are reviewed with the project manager, project sponsor, executive sponsor, and the PMCoE. The PMCoE briefs the CTO on these quality assurance findings. The findings may include a recommendation for continuing the current level of oversight, or adjusting the level of oversight up or down.

Formal Quality Assurance

Projects assessed at the highest levels of inherent risk are required to have formal quality assurance throughout the project. An independent quality assurance consultant is assigned to the project to provide continuous review throughout the project life cycle. The quality assurance consultant works closely with the project sponsor, steering committee, and project manager, focusing on the project management processes applied, and monitoring identified risks and risk mitigation plans. The PMCoE attends Steering Committee meetings where QA reports are presented and may escalate issues if recommendations for addressing significant risks are not making progress. The goal of oversight on these projects is to ensure early identification of signs of unsatisfactory performance so that corrective action plans can be promptly developed in consultation with the executive sponsor and the CTO.

IT Project Risk Profiling in the City of Seattle

A key challenge for the PMCoE in implementing their strategic action framework was to gain buy-in from the local departments for the various levels of project oversight. Departments sought to maintain their autonomy, and since the formal level of oversight, in particular, is an added overhead for a project, there was considerable reluctance among some departments, who questioned the value of the oversight contribution. Although the PMCoE had a clear mandate from the CTO to deliver demonstrated improvements in the city’s IT project management, it chose to implement the proposed changes to project risk profiling and monitoring collaboratively, seeking to work cooperatively with local department staff to achieve the desired outcomes.

The collaborative risk profiling process begins at the project initiation stage. At this point PMCoE staff work with the local project manager to complete a questionnaire outlining the attributes of the various dimensions of the proposed project. Together, the PMCoE and the local project manager map the measures of the attributes on a risk spider chart (see Figure 2 for an example of a completed chart). The overall pattern of the attributes displayed on the chart helps to determine the risk profile assessment for the project, which in turn determines the project’s recommended oversight level.

The risk spider chart shown in Figure 2 is an example of a project scenario — the development of a work management system — used in training sessions. An oversight level of formal quality assurance with monthly independent project quality assurance reviews would be recommended for this project, due to its size, business, and technical complexity, span of impact, and internal and external visibility. A detailed description of this project scenario’s characteristics is presented in Appendix A, Table A1, and Table A2 contains the resulting preliminary risk identification and preventive and contingent risk planning for the project.

The attributes identified on the risk spider chart provide a starting point for preventive and contingent risk management planning, and also offer suggestions for immediate actions that may help to reduce the risk of projects with high levels of inherent risk. For example, when a moderate or significant gap is identified between project manager experience and the nature of the project, the PMCoE can recommend a mentoring arrangement, or provide a project advisor to support the project manager. Similarly, projects with very long planned durations can be decomposed into a series of smaller projects, and projects with high uncertainty around requirements can be split into an initial needs analysis project so that the implementation project can be more accurately scoped and estimated.

The use of a graphical tool to visualize the nature of the project risks is an integral part of the PMCoE’s risk profiling strategy. In earlier iterations of the early risk profiling strategy, the PMCoE presented a narrative assessment of the project and encountered considerable resistance, because project managers sometimes challenged each individual part of the assessment, rather than focusing on the overall level of risk. With the graphical risk spider chart, even if there is still debate about some of the individual measures, the visual presentation of the overall inherent risk of the project is very impactful, and enables the PMCoE to move quickly to discussions about the best way to manage and monitor the project from an overall risk standpoint.

Figure 2: Risk spider chart for a work management system.

The risk spider chart approach, together with the monthly dashboard reporting has also enabled the PMCoE to make progress on its other mandated task of provision of project management consultative services and training. The chart and monthly reports provide a useful conversation starter around project risks and risk mitigation strategies, both with project managers and project sponsors, and enable the PMCoE to provide mentorship and guidance to managers and sponsors on their roles and responsibilities. During the review process throughout the project life cycle, PMCoE staff members meet regularly with the department project manager to discuss project status, issues, risks, and progress. In addition, the larger, more complex projects, that have been assigned an oversight status of formal quality assurance, also undergo oversight by an independent project oversight consultant, who conducts more in-depth monthly reviews of projects. At completion, each project submits a project closure report, which provides information and data on project outcomes such as: scope delivered, objectives achieved, solution quality, customer satisfaction, cost and schedule performance, and lessons learned.

There are many dynamics that impact project performance, such as technical environments, business environments, management cultures, project manager and sponsor experience, maturity of methodologies and processes, and user group and team dispersion, to name just a few. While the PMCoE recognizes the limitations around reliably determining the key contributors to period-to-period changes, the information harvested from the monthly project reports, oversight activities, and the project closure report help to inform insights on key contributors to the overall performance.

City of Seattle Project Metrics

The City of Seattle has been tracking its IT project performance improvement since the introduction of its project review and oversight initiatives in 2004. The monthly dashboard reporting on IT projects enables the city to track IT project performance against budget and schedule. Project performance in 2006 was used as the original benchmark, with 57% of the 14 completed projects that year achieving less than 10% cost variance and 36% achieving less than 20% schedule variance. For the period from 2006 to 2011 a total of 76 projects were completed, and Seattle saw a steady improvement in performance. Project completion data as of 31 December 2011 show that, of these 76 projects, 75% achieved less than 10% cost variance and 46% achieved less than 20% schedule slippage. However, the tracking approach used during this period was to measure actual cost and schedule performance “at completion” against the estimates made at “initiation” when the project was chartered. Schedule and budget estimates at initiation stage carry high levels of uncertainty and can often be unrealistic, and there was some concern that the performance improvements may have been over-stated.

Thus, the PMCoE refined the tracking measurements to provide more accurate data by clearly establishing and clarifying the baseline benchmarks, and the tracking metrics were changed to capture actual cost and duration of a project “at completion” measured against the estimates developed from “planning stage” activities. If the project uses rolling wave planning, the first estimate is expected to be sufficiently realistic for completing all remaining phases. The revised performance tracking was initiated at the start of 2011 and there is now a base of 27 projects in the data. The current cost and schedule performance data as of 31 May 2013 are as follows:

Cost Performance

o81% of projects were completed below or within < 10% variance to the planning phase budget, showing a 5 point improvement from the 76% at this time last year.

o15% were completed between 10% and 20% of planned cost estimate, representing a 3 point improvement from the 18% one year ago.

oNo projects exceeded 20% of planned costs set at the planning stage baseline.

o4% were completed with an undetermined variance to planned costs, down 2 points from the 6% one year ago.

Schedule Performance

o11% of projects were completed below or within < 10% variance to the planning phase duration, showing a 1 point improvement from the 12% one year ago.

o30% were completed between 10% and 20% of planned schedule estimate, which is down 5 points from the 35% one year ago.

o56% were completed with variance in excess of 20% of the planned schedule estimates, which is an increase of 9 points from the prior year’s 47%.

o4% were completed with an undetermined variance to planned schedule, down 2 points from the 6% one year ago.

Staffing constraints formed the major challenge for schedule performance, due in part to tightened staffing policies during the recession. In particular, some of the longer projects suffered reduced priority for resources as new higher priority projects kicked off. The PMCoE expects cost performance to continue to improve as a result of their on-going mentoring and training of department project managers and the provision of improved tools, methodologies, and support. Additionally, as the economy improves, staffing issues should begin to stabilize.

In 2011, the PMCoE also began collecting information on scope performance, in particular, scope elements planned versus those actually delivered, the percentage of business and technical objectives achieved at the close of the project, and opinions on the quality of the solution and the level of user satisfaction. The goal with these measures was to begin to provide a basis for the evaluation of system benefits and to add emphases to these project attributes when the project leadership assesses project outcomes. A positive outcome already observed is that teams are beginning to learn that ambiguity in the specification of objectives developed early in the project makes it very difficult to accurately measure achievement at completion.

Conclusion

The PMCoE’s risk profiling and oversight process provides a visual aid for surfacing overall inherent risk at the early stages of IT projects, thereby enabling the proactive implementation of appropriate management strategies before problems strike. The PMCoE’s risk assessment process, and the risk spider chart that is the primary tool in this assessment, involves a review of a small set of project dimensions known to be related to project performance, rather than the detailed checklist of a large number of potential risk factors typically used in traditional risk assessments. The dimension-based assessment provides an easily measurable determination of the overall risk level of a project, without requiring probability-impact estimates of individual risks, which often amount to little more than guesswork. The tool has now been used on over 100 projects and the organization has been able to demonstrate a steady improvement in project metrics since 2006 when the work began. As with many organizations, the City of Seattle is continually working to achieve further process improvements; the risk profiling and oversight procedures described here have undergone, and will continue to undergo, refinements to fine-tune and further streamline the process. However, the work carried out by the PMCoE over the past 10 years has provided a solid foundation to support ongoing incremental development and improvement in their IT project risk management processes.

The City of Seattle’s project risk assessment process is a model for other organizations striving to engage in effective practices to improve IT project outcomes. The tool is readily customizable for different organizations and project contexts. A key outcome has been the insight gained into overcoming the practical barriers that may have hindered the implementation of extensive IT risk management research findings in practice.

References

Bannerman, P. L. (2008). Risk and risk management in software projects: A reassessment. Journal of Systems and Software, 81(12), 2118–2133.

de Bakker, K., Boonstra, A., & Wortmann, H. (2010). Does risk management contribute to IT project success? A meta-analysis of empirical evidence. International Journal of Project Management, 28(5), 493–503.

Herbsleb, J., Zubrow, D., Goldenson, D., Hayes, W., & Paulk, M. (1997). Software quality and the capability maturity model. Communications of the ACM, 40(6), 30–40.

Howell, D., Windahl, C., & Seidel, R. (2010). A project contingency framework based on uncertainty and its consequences. International Journal of Project Management, 28(3), 256–264.

Jiang, J. J., Klein, G., Hwang, H.-G., Huang, J., & Hung, S. Y. (2004). An exploration of the relationship between software development process maturity and project performance. Information & Management, 41(3), 29–288.

Martin, N. L., Pearson, J. M., & Furumo, K. (2007). IS project management: Size, practices and the project management office. Journal of Computer Information Systems, 47(4), 52–60.

McConnell, S. (2006). Software estimation: Demystifying the black art. Redmond, WA: Microsoft Press.

Sambamurthy, V., & Zmud, R. W. (1999). Arrangements for information technology governance: A theory of multiple contingencies. MIS Quarterly, 23(2), 261–290.

Sauer, C., Gemino, A., & Reich, B. H. (2007). The impact of size and volatility on IT project performance. Communications of the ACM, 50(11), 79–84.

Schmidt, R., Lyytinen, K., Keil, M., & Cule, P. (2001). Identifying software project risks: An international Delphi study. Journal of Management Information Systems, 17(4), 5–36.

Shenhar, A. J. (2001). One size does not fit all projects: Exploring classical contingency domains. Management Science, 47(3), 394–414.

Shenhar, A. J., Dvir, D., Levy, O., & Maltz, A. C. (2001). Project success: A multidimensional strategic concept. Long Range Planning, 34(6), 699–725.

Sommer, S. C., & Loch, C. H. (2004). Selectionism and learning in projects with complexity and unforeseeable uncertainty. Management Science, 50(10), 1334–1347.

Standish Group. (2005). Chaos rising. West Yarmouth, MA.

Subramanian, G. H., Jiang, J. J., & Klein, G. (2007). Software quality and IS project performance improvements from software development process maturity and IS implementation strategies. Journal of Systems and Software, 80(4), 616–627.

Wallace, L., & Keil, M. (2004). Software project risks and their effect on outcomes. Communications of the ACM, 47(4), 68–73.

Wallace, L., Keil, M., & Rai, A. (2004). How software project risk affects project performance: An investigation of the dimensions of risk and an exploratory model. Decision Sciences, 35(2), 289–321.

Appendix A: Example Project Description, Risk Spider Chart, Oversight Recommendation, and Preliminary Risk Planning

Table A1: Typical City Project Scenario: Work Management System.

Description	Work management is essential to maintenance and other field activities performed by the department. Through work management, crews are scheduled, costs are tracked, billing information is created, asset tracking systems are updated, and management receives critical reporting information. Currently, work management activities are supported by a variety of separate division-level information systems, user-developed tools, and/or paper-based processes. This makes unified reporting virtually impossible, increases system maintenance costs, and precludes a comprehensive view of the day-to-day work-related activities. The planned work management system will standardize departmental workflow processes where feasible; allow for better tracking of work activity, costs and schedules; and provide more consistent data for management reporting.
Criticality	Work management is a mission critical application, directly impacting the ability to efficiently direct the flow of work, track expenditures, monitor work progress, coordinate the effort of multiple workgroups, and support asset management.
Scope	The work management project will implement new functionality replacing identified legacy systems and standardizing work management business practices. Some level of discovery is required to determine how to converge the new technology with standardized business processes to achieve optimum results for department work units. Yet to be determined is what functions will be deployed to field crews.
Deliverables	The implementation of a customized off the shelf software (COTS) work management system for four divisions, including detailed cost tracking and document management: •New/redesigned business processes: Standardized business processes across all divisions to facilitate implementation of work management. •Data: Establish data exchange standards across all work management related systems; convert all existing work management data in existing applications; convert open work orders; ensure manual accessibility for historical work orders. •Interfaces: Automated interfaces to and/or from all external data sources required, including: oHuman resource information system (HRIS) to record employee time and employee information. oInterface to download addresses from master address file. oInterface to accounting system to download project and activity budget codes and for billing specific fees. oInterface to fleets and facilities vehicle information. •Implement document management as a tool within the work management system to manage images and paper record specific to work management. •Provide technical training for project team on new technology; user training for each division’s unique work groups on business procedures and system use; establish a user support team. •Provide an operations and support plan; support documentation; help desk training; and business continuity plan. •Decommission three existing systems/applications.
Success Criteria	•The project delivers the mandatory features and functions to be defined during requirements. It is important to “get it right,” budget and schedule constraints notwithstanding. •The project completes within the approved budget range, to be established during the planning phase of this project. •The project is completed by the baseline completion date, to be established during the planning phase of this project.
Project Sponsor	The sponsor and business owner’s prior experience is limited for this scope of system. He or she has sponsored small plug and play application implementations and has a low tolerance for risk and prioritizes budget over scope, scope over schedule.
Project Manager	The project manager has limited experience with this size and complexity of project.
Approach	Due to the interrelated nature of business processes and interfaces the plan is to implement all divisions in a single release.
Schedule	Estimates are based on a similar project in another jurisdiction: •3 months allocated for initial planning •1 month allocated for procurement •2 months allocated for joint department/vendor detailed planning •24 months for implementation •Estimated 30 months total duration plus contingency.
Cost	•Estimates to-date based on a similar project in another jurisdiction. •Total project cost will be established at the end of planning but the not-to-exceed budget amount is US$3.5 million, including contingencies.
Technology	•The core technology is anticipated to be one of the top tier work management software solutions. The selected technology must be proven in the industry, but may be new to the city and the department. •The solution will require integration of a work management application, a business rules application, reporting engine, and a mobile application. Significant configuration is anticipated and some customization may be required. •A significant amount of data will need to be migrated from the applications that will be replaced. Data cleanup will also require considerable effort. •Six interfaces of moderate to high complexity will need to be reworked. •Laptops or other mobile devices will provide field personnel with real-time access to the work management system. Field personnel are not current users of such technology. •The application must be available 7 X 24 X 365 to support work crews operating around the clock during critical situations.
Business Complexity	The project will impact users and business processes in the owning department and two of the city’s largest departments. In addition, a number of external agencies that perform work on a contract basis will be required to use the new application and comply with new policies and procedures. •Business processes are not fully documented and must be redesigned to streamline and standardize department operations. •Business users are accustomed to having technology customized to meet business needs that do not align with the native functionality of the work management system. •Business process redesign is expected to result in changes in staff roles and job requirements for represented and non-represented personnel. •Department personnel will require training on the new system and business processes that are tailored to their unique job requirements. •The work management system will need to align with new policies, rules, and processes that will be implemented through the financial standardization project.
Team	•About 25 business and technical personnel will be involved for the project. •The business team will be comprised of internal subject matter experts who will allocate a maximum of 20% of their time to the project. •The technical team will consist of internal IT staff with contract and vendor staff providing unique project-specific skills not available in-house. •Internal IT staff members are not currently familiar with the work management systems technology and will require technical training.
Vendors / Consultant	•Work management solutions are common in several industries and reports in trade journals suggest that providers of the most popular solutions are over committed. •Trade journals indicate there are several new vendors marketing work management solutions.
Methodologies	•Project management methodologies are documented, repeatable, and teams are familiar with them and accustomed to scaling to the complexity of the project. •There is no structured methodology for building or implementing COTS software applications. Projects typically defer to the vendors Software Development Life Cycle (SDLC) /Implementation processes. •Configuration management processes are unstructured.
Visibility	•The project is on the department director’s accountability agreement and is of great interest to city council members and the mayor’s office. •Recent news articles suggest the media may have a keen interest in the outcome of the project.
Users	•1,000 users across all impacted organizations. •Users are accustomed to loosely structure manual processes. •Field personnel have little on-the-job experience using online applications. •Most users are represented by a labor agreement.
Assumptions	The successful completion of this project is based on the following assumptions: •This project is constrained by budget. It is anticipated that scope will need to be constrained for the project to be delivered within the final budget target. •A COTS solution will provide a workable solution for work management. •A request for proposal (RFP) process will be used to select the best technology for work management. •After requirements are gathered, a gap-fit analysis will be performed against each candidate product. •The city’s existing technology infrastructure will support the work management solution. •Resources and subject matter experts will be available when required. •The project sponsor has final approval authority for all project deliverables. •This project is a top departmental priority.
Constraints	•Features to be included within the scope of the project are available in the selected COTS application. •Solution must comply with city technology standards and business practices. •Total costs not-to-exceed budget amount is US$3.5 million. •Project must be fully implemented by September 2014. •Internal IT resources are constrained by the department’s information systems (IS) governance group’s allocation of internal IS resources to the project. •Hardware, software, applications, and services will be procured using standard city procurement processes and procedures. Multiple providers are anticipated.
Dependencies	•Work management must align with the asset management program and contribute data collection to the asset management decision-making processes. •The Oracle Upgrade Project must be completed prior to the planning phase.

Table A2: Preliminary Risk Management Recommendations for a Work Management System Project Scenario
[This table presents a partial analysis for illustrative purposes only]

Attribute	Risk	Preventive Action	Contingent Plan
Criticality	Council resolution and audit requirement increase likelihood of internal and external scrutiny and negative press if the project becomes challenged	Engage independent QA to help identify risks and keep project on track.	Ensure communication plan addresses the potential for negative press and measures to keep executives and media informed.
Visibility	High visibility with the mayor, council, and media may increase exposure of project problems internally and negative press externally.	Apply strong project disciplines to decrease likelihood of problems. Include public information officer in the steering committee. Ensure communication plan includes frequent council briefings on progress and problems.
Scope Uncertainty	The work required to converge the new technology with standardized processes and uncertainty about functions to be deployed to field crews is not clear and therefore cannot be planned with any accuracy. Document management implementation in parallel with work management adds complexity that may decrease the odds of success.	Acquire the services of knowledgeable experts to advise the planning effort. Confer with an organization that has implemented the technology in a similar setting to inform planning and estimating. Contract with an experienced provider to implement the solution. Work with the sponsor to clarify scope in these areas.	Add appropriate cost and schedule contingencies to estimates to accommodate discovery and rework.
Technology Uncertainty	Proposed technical environments may not be achievable within the constraints of the current infrastructure.	Work with the enterprise architect to understand constraints within the technical environment and options for implemented desired solutions.
Business Process Changes	Standardizing business processes is difficult across unit boundaries. Due to the magnitude of the business process changes, there are likely to be a number of users who are resistant to the changes.	Develop a governance structure that includes a business change team with representatives from each organization. Develop a comprehensive cultural change management plan.
Customization	Strategy to customize the application instead of changing business processes adds complexity and cost to the project and long-term maintenance of the application.	Work with sponsor to champion a strategy to adapt business processes to align with the native functionality of the application. Develop a total cost of ownership (TCO) model to show cost of downstream upgrades.	Ensure increased downstream costs are included in future budget proposals.
Data Conversion Complexity	Conversion of non-standard data is labor and time intensive.	Develop a data standard and implement a plan for data cleanup prior to conversion.
Application interface complexity	Technical complexity of integrating four systems with six interfaces may not be understood.	Conduct desktop exercises with a variety of subject matter experts to clarify the integration work.
Dependencies	Delays in the Oracle Upgrade Project will impact the work management schedule. Work required for alignment with asset management program is unclear and adds complexity.	Monitor Oracle Upgrade Project progress and work with sponsor to re-schedule if necessary. Acquire the services of experts knowledgeable about the asset management program early in the planning stages to advise the planning effort, and identify any roadblocks.
Span of Impact	Standardizing processes across multiple departments usually requires a lot of coordination and communication and it may be difficult to get agreement among all user groups. Reliance on external vendor may introduce over promising and under delivery, leading to quality issues and schedule delay. Vendor may assign inexperienced staff that may lead to quality issues and schedule delays.	Work with sponsor to champion the benefits of change with other executives and across the user population. Ensure vendor selection process carefully vets all vendors. Conduct rigorous reference checks on vendor. Ensure contract includes specific acceptance criteria for each deliverable. Base contract payment on deliverable acceptance. Include penalties for missed targets. Ensure the contract specifies experience qualifications of assigned staff.
Duration	Long duration projects are inherently risky. Getting agreement on all the standards, business processes and other changes requires considerable time that may result in a longer than planned schedule. Schedule estimates may not include adequate contingency because team is unfamiliar with the complexity of technical integration.	Decompose the project into a series of small projects that allow for quick wins and allow users to absorb change at a slower rate. Have executives communicate the reasons for the project and “what’s in it for you.” Build in extra time for testing.	If schedule variance exceeds 10% for three consecutive weeks, review project estimate at completion (EAC), and achievability of success criteria. Terminate the project if schedule variance exceeds 25% for two consecutive months and corrective actions do not indicate the project can meet schedule objectives to meet business justification for continuing.
Budget	Cost estimates are highly uncertain because they are not based on the actual work of this project. Cost estimates may not include adequate contingency because team is unfamiliar with the complexity of technical integration.	Decompose the project into reasonable work packages and activities and conduct a disciplined effort to accurately estimate project costs. Continue to refine the cost estimate as new information becomes available.	If budget variance exceeds 10% for three consecutive weeks, review project EAC and achievability of success criteria. Terminate the project if budget variance exceeds 25% for two consecutive months and corrective actions do not indicate the project can meet cost objectives to meet the business justification for continuing.
Team Size	Business team time constraints may lead to schedule delays. Lack of familiarity with technology will obscure vision of work, risks, and estimating.	Work the business sponsor and business team resource manager to secure commitments for time. Have sponsor communicate the real priority of the project relative to competing work. Acquire technical training for team members.
Methodologies	Reliance on vendor methodology could result in late discovery of issues, significant rework and increased cost, if the methodology is inadequate.	Adopt a disciplined methodology for implementing the COTS application and train all team members on the methodology. Review methodologies and agree on how internal and vendor methodologies will converge.
Project Manager	Project manager not experienced with projects of this scope and complexity.	Seek a more experienced project manager. Provide a mentor.	Begin new job search.
Project Sponsor	Sponsor not experienced with IT projects of this size and complexity. Sponsor’s low tolerance for risk will require more detailed planning to achieve sufficient confidence levels to drive prompt decision making. Planning timeline will increase by X% and cost by Y%.	Review project critical success factors and project risks with the sponsor and propose a sponsor advisor to support sponsor. Work with the sponsor to develop very specific roles and responsibilities within the governance structure.
Users	Large user population with variability of user training requirements may require increased investment in training planning and delivery.	Develop a comprehensive training plan based on training needs analysis to ensure all user audiences are appropriately trained.

Hazel Taylor is an Associate Professor at the Information School, University of Washington, Seattle, Washington, USA; she holds a PhD from Queensland University of Technology, Brisbane, Australia, and prior to joining the Information School, she taught at the University of Waikato in New Zealand and at the Hong Kong University of Science and Technology. Her teaching and research focus on IT project management and risk management with an emphasis on tacit knowledge and decision making. Prior to her academic career, she worked in industry with manufacturing, construction, and government organizations, both as a systems manager and an IT project manager.
Ed Artman led the City of Seattle’s Information Technology Project Management Center of Excellence, which oversaw large complex information systems projects across the city on behalf of the Chief Technology Officer. He is a Project Management Professional (PMP)® credential holder and certified ScrumMaster with over 20 years’ experience in IT project management. As a passionate practitioner of project management best practices, Mr. Artman has successfully managed a wide variety of business and technology projects, and has extensive experience in recovering troubled IT projects. His industry experience includes the fields of technology, distribution, retail, transportation, insurance, real estate, utilities, hospitality, and government.

This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI or any listed author.

Early risk assessment in IT projects

integrating risk research into project management practice

Information School, University of Washington, Seattle, Washington, USA

Edward Artman, PMP

Department of Information Technology, City of Seattle, Seattle, Washington, USA