Improving risk management outcomes during I.T. led business change projects
case studies in the Australian workers compensation industry
Despite significant research efforts, many IT projects continue to be abandoned or do not achieve their initial goals. It seems practitioners who implement IT projects must either be unaware of the guidance available, or if aware of the guidance are unable to implement it.
A recently completed thesis explores this topic through a number of in-depth case studies (Bailey, 2004).
This presentation will examine the apparent disconnect between the plethora of advice purported to solve the project failure problem, and its continued occurrence.
IT Project Failure
Some examples of institutions and standards established to improve practice and project outcomes:
- Software Engineering Institute (SEI). Formed in 1984 by the US Department of Defence in response to IT projects consistently overrunning budgets and schedules, or failing to complete
- PRINCE (Projects IN Controlled Environments). In 1989, CCTA (UK) developed this IT project management methodology following an unacceptable level of IT project failure in UK government projects
- A Guide to the Project Management Body of Knowledge (PMBOK® Guide) First published in 1996 by the Project Management Institute (PMI).
Yet, the Standish Group in 2004 found that 18% of all IT application development projects in the US are abandoned altogether, and a further 53% of projects failed to meet cost or benefit expectations. Standish Group also estimates that failed projects cost the US economy $US75 billion per annum.
Mary researchers have expressed views as to the root causes of IT project failure and advised how best avoid it. There is disagreement regarding relative importance of reasons for failure, as well as conflicting advice on steps to avoid it.
Past research investigating specific types of IT project failure has come up with strikingly similar findings:
- Business Process Reengineering Project Failure. (Davenport, 1993) found that 70% of BPR projects were ending in failure
- Customer Relationship Management Project Failure. An estimated 65% of CRM implementation fail to complete (Davids, 1999)
- Knowledge Management Project Failure. 70% of KM projects don't accomplish what they set out to do (Ambrosio, 2000).
Case Study Design
Almost exclusively, IT project failure research assesses reasons for failure based on surveys or interviews conducted AFTER the failure outcome was known. The approach decided here was to conduct in-depth analysis of projects while they were underway. The Grounded Theory approach informed this study.
The Australian workers compensation (WC) insurance industry was selected. In Australia, high-profile mergers and acquisitions, demutualisation and corporate failures characterise this industry. Four of the top five WC managed fund insurers in 1998 has subsequently been taken over or ceased operations. At the time of the research, WC was in crisis, and the NSW Government had launched the Grellman Enquiry to recommend ways to manage $A2Billion of unfunded liabilities.
The research was conducted following agreement by two WC companies to be studied (InsurCo1 and InsurCo2). Each offered two significant projects for review, one a successful workflow/image implementation, and the other less successful. One case had significantly more data available to review than the other three, and was selected as the main case.
Main Case Study Description
InsurCo1 is a major workers compensation insurer, in both underwritten and managed fund sectors, operating Australia-wide. The case study concerns a significant IT implementation within a major BPR project. Timeline of events leading up to the project:
- An evolutionary change program commenced mid 1997 “Genesis”. This took a year and delivered modest work practice improvements, insufficient to meet the scale of reform required for profitability
- A new NSW WC Manager was appointed following predecessor promotion. This individual became the champion that drove the workplace change program
- InsurCo1 had decentralised workers compensation operations to 11 Regional offices, which had mainframe-connected “green screens”
To manage risk of a more radical change program, the new manager commissioned an analysis project in conjunction with an external consulting firm. “Before2000”, with $A1M budget, commenced November 1997, asked to report by June 1998:
- Ten staff from Regional offices and the Genesis PM seconded
- Compared “as is” operations with multiple “to be” scenarios: degrees of centralisation and IT investment
- Recommended: A major BPR project to implement workflow and image technologies, and consolidate to 3 locations. Transition to modern PC's, with high-resolution image-capable screens
- Estimated budget to complete BPR: $A12M, timeframe 18 months
The Grellman Report was released, recommending immediate move to mandate early intervention and injury management effective September 1998. NSW WorkCover changed insurer remuneration to encourage focus on these requirements.
This led InsurCo1 to delay the BPR project for four months while implementing the requirements of the Injury Management act:
- The Before2000 PM was seconded full-time to the Grellman project
- Urgency required single Injury Management Centre, in Parramatta to meet September deadline. Second IMC in Newcastle planned later that year
- Internal job advertisements invited staff to transfer to the IMC
- The 11 regions to continue processing existing tail claims
The lure of new office environment and modern PC technology meant the IMC was fully resourced, with a significant number of these being ex-Parramatta Region office staff.
The new WC Manager had focused totally on ensuring IMC project success, however a “meltdown” in the Parramatta Region office occurred, where insufficient skills remained to continue tail claims processing. A series of arguments with the GM about the problems led to the new manager being exited from InsurCo1.
A replacement WC Manager was appointed from the General Insurance division of InsurCo1. This was the second WC Mgr in less than 12 months with no prior operational experience in WC insurance business. The replacement manager remained committed to the Before2000 recommendations, but went further to announce a vision of achieving “world class claims management”. An experienced PM was assigned and workflow vendor selected. Visits to two other Australian Insurers that had recently undertaken similar projects were organised. One of these was InsurCo2. This gave increased confidence to proceed.
The WCCM project kicked off with great fanfare under a Project Director with 5 workstreams and 35 full time personnel. The re-engineering of work practices and implementation of a pilot claims operation was budgeted to cost $A4.5M. IT workstream scope was workflow/image automation to integrate with existing mainframe claims administration system.
The workflow vendor had just released WKFlowMK2 worldwide, and was developing a new fast-track tool, RAPID. Workflow rules were developed by the work practice workstream, and interface programs written to integrate WKFlowMK2 with the mainframe administration system. However, problems began emerging when WKFlowMK2's RAPID software was well behind schedule.
When the RAPID software was finally released to the InsurCo1 project team, multiple software bugs were encountered, even after many of these were resolved, there continued to be critical performance problems. Despite intense efforts, WKFlowMK2 and RAPID software could not pass even the pilot load test. Transaction response times were almost 2 minutes, even with only 35 users (full requirement of over 400 concurrent users).
Rising concern by the executive led to a project audit being commissioned by InsurCo1's PMO. This reviewed the whole WCCM program, and recommended:
- Realign goal of “world class”. How can this be measured anyway?
- The complexity of 5 streams and 35 full time consultants too big. Reduce to 3 more focused workstreams
- Pause IT development, and re-visit formal business case to decide what reengineering and IT should be implemented.
Further deterioration in WorkCover revenues and overall profitability required the InsurCo1 to reduce operational budgets by $A5Mp.a. As well, the GM announced the immediate downsizing of the WCCM program. The Program Director resigned in protest.
The grand vision of the replacement WC Manager for WCCM was pared back to “workflow claims management”, WCM. This was limited to 2 workstreams: BPR improvements that could deliver immediate bottom line improvements; and supporting workflow/image IT implementation.
Replacement workflow software was evaluated. Following competitive assessment, Worksoft97 was selected. Given past problems, software vendors required to work together to deliver a fixed-price proof of functionality and load capability, with payment on completion. The re-visited business case predicted $A23M ROI over 5 years. The Steering Committee gave approval to continue, with strong personal support from the WC Manager.
Integration of Worksoft97 with the imaging and mainframe claims administration systems went smoothly. The deployment to the pilot group went without a hitch, and the load tests revealed excellent response times, even at loads of 150% of Australia-wide deployment. Claimants and providers recorded high satisfaction with the quality and speed of the claims being administered by the pilot team.
External events: InsurCo1 had been in takeover implementation with a new acquirer for the previous six months, however WC had been left alone, as the acquirer did not have a significant WC business, hence little integration was required. A vacancy in the merged IT Executive was successfully sought by the NSW WC manager, who left this key role after only 14 months in the job. The GM subsequently restructured his management team nationally, thus not directly replacing this vacancy.
The new merged company CFO called for full ROI on all planned and underway projects, including the WCM program. These were required to be submitted in writing, and subjected to face-to-face review. The new National WC Manager, not particularly involved in the WCCM or WCM programs directly, did not particularly press for the program to proceed, despite the strong written TCO, which supported it. The capital investment requirement of $A9M to roll this out in NSW was eventually to seal it's fate.
On instruction from the CFO, the PM closed the project effective immediately. The imaged documents were printed out and placed back into paper claims files, pilot team dispersed back to their old jobs and all saleable hardware and software disposed of.
Main Case Study Analysis
The primary data source was the content analysis of the case study firms' internal documentation and published literature. This was over 2100 pages of documentation, in around 240 memoranda, electronic mail messages, letters, business proposals and meeting minutes. The data was entered into N-Vivo©, and the coding activities appropriate for Grounded Theory performed. Initial insights led to semi-structured interviews with case participants. The General Manager, IT Manager and IT Project Manager were all willing to be interviewed. These interviews provided valuable comparison to the documentary evidence.
Following the Grounded Theory approach, the case data provided a number of initial perspectives by which the data might be assessed:
- Project Risk Awareness. To what extent did the participants demonstrate that they were aware of “best practice” in avoiding problems that might lead to failure? It seemed that practitioner project methodologies represented a definitive source of wisdom. The particular PM on the main case study was a keen advocate of PRINCE2, for example. I decided to adopt the PMI's A Guide to the Project Management Body of Knowledge (PMBOK® Guide) (2000) Risk Management chapter as representing definition of best practice knowledge at the time of the projects under study. A Guide to the Project Management Body of Knowledge (PMBOK® Guide) described four classifications of risk, broken down into a total of 15 types of risk. The case analysis looked at each type of risk, and questioned:
– Was there evidence of presence of that particular risk?
– Any evidence that mitigation of that risk was attempted?
– Did that risk area impact the project?
- Personnel Turnover. The NSW WC Manager role had a number of changes. Was this a feature of this and other cases? Was there evidence of a causal link?
- Key People Issues. The issues of Business Ownership, the Role of Consultants and level of commitment to future projects were all themes evident in the case data.
A summary of Project Risk Awareness analysis is shown as Exhibit 1.
Exhibit 1 – InsurCo1 Project Risk Analysis
Some examples of the observations from this table are:
- Project Management category has three risk areas – all indicating a low evidence of presence, and a high evidence of mitigation. The formal project documentation appeared to represent sound practice. Hence, impact of these risks coded as none or low
- Unproven Technology Risk shows high presence, yet high in recognition and attempts to mitigate. In the end, however, this appears to have had high impact on the project outcome
- Merger & Acquisition Risk. Awareness of M&A activity is documented in project reports, and interaction with the M&A team evidences mitigation attempts. In the end, impact was high, as the project abandonment could be significantly attributed to the merger priorities of InsurCo1's business.
Staff Turnover was significant, and is graphed in Exhibit 2.
Exhibit 2 – InsurCo1 Personnel Turnover Analysis
There is significant evidence of commitment to the project varying over time. The final outcome was the 3-times removed WC Manager not backing the WCM project to the CFO, leading to the executive decision to abandon it. While no causal relationship could be confirmed, it appears that personnel turnover created risk for this project.
The people issues dimensions:
- Ownership - High level of commitment and involvement by Business and IT Executives. Later interviews highlighted the project's PM perceived a lack of commitment by the GM as partly the cause of the project's demise
- Role of Consultants - This went through waves of positive and negative indications in the real-time case documentation. Very positive role of the separate Genesis and Before2000 business case consultants was to be replaced with strongly negative emails and minutes regarding vendor consultants not committed to InsurCo1 outcomes.
- Commitment to subsequent projects - Was at an all time low at the abandonment time of the project, and even a year later when the interviews were conducted.
Supplementary Case Studies & Cross-Case Analysis
The process described above was repeated for the three supplementary case studies. Briefly, these cases were as follows:
- InsurCo1 also undertook an implementation of Image processing of all NSW Policy documentation. The policy files were going to take half the entire floor space of the newly-centralised Policy department, following Genesis recommendations to consolidate into a single location. The business case to undertake the project was a one-page email from the new NSW WC Manager to the GM in March 1998. The decision was made 24 hours later, and an external contractor with previous Image system implementation experience hired. The project completed in less than 3 months, with no major issues or personnel turnover.
- InsurCo2 was a competitor to InsurCo1. Between August 1996 and July 1998, they undertook a major BPR of their entire organisation, “New Shape”, part of which was an Australia-wide implementation of workflow and image technologies for their WC business. The implementation in NSW had uncanny similarities to the InsurCo1 project: centralisation of many offices into 2 or 3 in each State; initial implementation planned for Parramatta; major performance issues with the workflow software proving the system was unable to properly support the full operations; turnover of some key stakeholder positions; etc. The software chosen was a different vendor that either of the InsurCo1 examples, yet stability and response time problems also plagued this implementation. Unlike InsurCo1, the CEO commitment demanded retention of the system, and staff were forced to develop many survival strategies and workarounds to get work done.
- A year after completion of the New Shape program InsurCo2, With Y2K pending, had not yet successfully implemented the new WC administration system that was going to replace their old one. No Y2K remediation had been undertaken on the old system. Crisis meetings reviewed the position, and they commenced Y2K remediation on the old mainframe system, and also scheduled a full re-implementation of their workflow and imaging environment. The extensive (and expensive) customisation made to the workflow software during the New Shape project was discarded, and a much less customised implementation of the software “out of the box” planned. Christened “Vanilla”, this bland deployment was to successfully resolve the horrendous performance and stability problems that business users had been suffering for 18 months.
Combined analysis of all four cases provides this research with the following overall findings:
Successful Projects common features
Project Risk Awareness analysis: Exhibit 3 confirms many impact scores low or none. A number of mitigation scores high, demonstrating high risk management efforts.
Exhibit 3 – Comparison of Successful Projects
Personnel Turnover: Key personnel change precipitated the project initiative. Stability of key personnel roles. Key People Issues: Both business ownership & commitment to future projects = High View of consultants: InsurCo1 – High. InsurCo2 – after New Shape, Low; by project end, High.
BPR and technology worked, and users happy with the new practices and IT. Reasons for success given:
- The PM was good – we should use him for all future projects !
- Short, informal business case
Bold, short timeframe characterised success
GM InsurCo1 and IT Manager InsurCo2 acknowledged success might be as much good luck as good management
Failed Projects Common Features
- Large Scale BPR, and first Workflow automation for each firm
- Both moving from mainframe screens to PC's on the desktop
- Large scale investment for the firms involved ($A15M and $A25M)
- Part of a larger BPR program throughout the firm
- Multiple-year project
Commonality of solutions selected:
- Decision to split claims processing
- Centralisation choice – both chose to start with deployment to Parramatta, then later in Newcastle
- Both projects conceived when a new executive sponsor hired
- Large Project Teams, secondment of full-time user staff, multiple concurrent streams
- Long lead time and extensive business case. Long delay between business case and start
- Application performance, functionality and stability problems with new, unstable software
Interestingly, InsurCo1 had full visibility to the InsurCo2 problems.
Project Risk Awareness analysis shown in Exhibit 4.
Exhibit 4 – Comparison of Failed Projects
Unproven Technology and Complex Technology had both High Presence levels, and High Impact.
Personnel Turnover: InsurCo1 – change in personnel led to abandonment. InsurCo2 – no change in key roles led to perseverance.
Key People Issues – All changed as the project progressed: Business ownership. InsurCo1 had NSW WC Manager as champion, but active distance by 4th incumbent. InsurCo2 had CEO Champion, but as things got tough, disillusioned middle management. Role of Consultants was high at the start, but threats of legal action by project's end. Commitment to subsequent projects. InsurCo1 was high at commencement (commenced Policy Imaging project), but very low be end of WCM. Insurco2 was high at start (commenced Data Warehouse project), but then went 2 years with no major new IT project and the rewrite of claims administration application was cancelled.
P1: Practitioners are unaware of guidance, therefore past failures continue. Judged FALSE
P2: Practitioners are aware of guidance, but fail to achieve success: Judged TRUE, some reasons:
- Guidance advises to break up large projects into smaller ones. Changed work practices, locations, IT capabilities all needed to be done together
- Guidance advises to ensure top management commitment. How to cope with personnel change, not just attitude change?
Findings not generalisable. Thesis did not attempt statistical validity.
Criticism of “avoidance advice industry”. This Thesis is yet another example!!
Implications for Practice
- Large and long-running projects have higher probability of failure
- Risk Management Factors are demonstrably present in projects, yet recognition and mitigation attempts alone not sufficient to assure success
- Top management commitment is key
- Avoid overly-detailed analysis at initiation
- If delays occur, reconfirm business case and commitment levels
- Keep business users involved, but not exclusively on the project
- Management of consultants/contractors is key
- Findings during projects yields different results than post-project assessments
- Challenged projects don't appear to just fail due to a single cause. Tow failure cases here had multiple risk issues, many not identified by post-project interviews.
Current IT Project Failure Trends
Further advances in project risk management advice, e.g. A Guide to the Project Management Body of Knowledge (PMBOK® Guide) (2004)
Introduction of Risk Management Standards, e.g. AS4360:2004
Focus on Corporate Governance and Project Governance Standards e.g. AS8015:2005
Project Portfolio Management and Earned Value techniques go beyond driving thru to meet initial goals; but assert that competition for investment funds should precipitate project closures if relative returns are eclipsed by more beneficial alternatives
Ambrosio, J. (2000) Knowledge Management Mistakes, Computerworld, Issue No. 27, pp 44
Bailey, P.J. (2004) Achieving Successful Implementation of Information Technology Led Business Change, Macquarie Graduate School of Management, Macquarie University, Sydney
Davenport, T. H. (1993) Process Innovation: Reengineering Work through Information Technology, Harvard Business School Press, Boston MA.
Davids, M. (1999) How to Avoid the 10 Biggest Mistakes in CRM, Journal of Business Strategy, Vol. 20, Issue 6, pp 22-26.
Project Management Institute (2000) A Guide to the Project Management Body of Knowledge – Second Edition (PMBOK® Guide). Newtown Square, PA: Project Management Institute
Project Management Institute (2004) A Guide to the Project Management Body of Knowledge – Third Edition (PMBOK® Guide). Newtown Square, PA: Project Management Institute
The Standish Group (2004), Chaos Demographics and Project Resolution – Third Quarter Research Report, Retrieved on 03/12/2005 http://www.standishgroup.com/quarterly_reports/pdf_copy/q3_intro.pdf
© 2005, Dr Philip J Bailey
Originally published as a part of 2006 PMI Global Congress Proceedings – Bangkok, Thailand