Abstract
The introduction of organisational project management maturity models such as the Project Management Institute (PMI®) Organizational Project Management Maturity Model Knowledge Foundation (OPM3®) model has provided organisations a means of assessing and improving their organisational project management capabilities. According to the theory, project management maturity models provide organisations with assessment frameworks, enable benchmarking, and indicate areas where the organisation can improve. This paper will examine whether these benefits do occur by examining some actual OPM3® cases from Australia, New Zealand, and the United States. The paper provides details of these assessments, including when, why, and how the assessments were undertaken and what benefits were obtained. Analysis of these case studies indicates that maturity assessments do provide organisations with benefits, including a structured assessment approach, benchmarking data, and improvement opportunities. The paper will also provide some lessons learned and some of the challenges and issues involved in undertaking maturity assessments.
Introduction
With the growing importance and interest in project management across all sectors of the economy, project management maturity models are important tools for practitioners, because they provide a focus on organisational capabilities rather than individual projects or project manager capabilities. One definition of project management maturity focuses on the degree to which an organisation practices systematic management of its projects, programs, and portfolios in line with its strategic goals, “… the application of knowledge, skills, tools and techniques to organizational and project activities to achieve the aims of an organization through projects” (PMI, 2003, p. xiii). That is, the degree to which the organisation does the “right” projects and does them “right,” or whether the projects selected for the corporate portfolio are in line with the organisation's long-term goals, constraints, and direction, and whether the projects are managed in a systematic method. It includes the management of individual projects (project management), groups of projects managed in a co-ordinated way (program management), and projects and/or programs managed to meet strategic objectives (portfolio management). There are other, broader definitions (Andersen & Jessen, 2003), which include behaviour and competence, and focus on “receptivity” to project management rather than “action.” However, this paper will use the former definition and focus on the OPM3® model.
Maturity models such as OPM3® can provide organisations a number of potential advantages and disadvantages. These advantages or benefits include an assessment framework that enables an organisation to compare its project delivery with best practice or against it competitors, ultimately defining a structured route to improvement (Pennypacker & Grant, 2003). That is, they enable organisations to do three things. Firstly, make an internal assessment of their project management capabilities, both strengths and weaknesses. This assessment is based on a framework that has been independently developed over many years by industry experts and practicing project managers. It provides a more independent, objective assessment. Secondly, it allows organisations to undertake benchmarking. This can be internal, between organisational elements, or external, against organisational peers or industry norms. Thirdly, it facilitates improvement initiatives, as organisations can then determine improvement paths based on the assessment and benchmarking data obtained. Project management maturity encompasses a broad range of organisational elements that underpin the organisation's project management capabilities. Project management practices, operations, processes, knowledge areas, competencies, and phases are all among dimensions of project management capability (Pennypacker & Grant, 2003).
However, there are some potential disadvantages with maturity models. Maturity models are based on total quality management (TQM) principles. Some key principles (for example, commitment and employee engagement) have not been accorded the importance in maturity models the TQM philosophy intended, the empirical track record of TQM is mixed, and the principles are complex to operate and interpret (Mitchell, Linger, & Owen, 2007).
OPM3® Overview
OPM3® (PMI, 2003 & Bourne, 2006) has existed for a number of years now, and as such, this section will provide a brief overview and explanation of the model. This is required to aid understanding of the model and as a prelude to the following section, where actual OPM3® cases will be reviewed. Released in 2003, OPM3® was developed over a period of about 6 years by over 800 volunteer project management practitioners, across all industries and disciplines, in some 35 countries. Consultants and practising project managers were involved in phases of the model, including both the research and design phases. Potential users of the model were surveyed in order to understand their requirements, which were then built into the model. A second edition, which will review and enhance the current version, is due for release in 2008.
The OPM3® model (PMI, 2003) comprises a number of interrelated elements, including knowledge, assessment, and improvement. The knowledge element simply refers to the user becoming comfortable with the contents and concepts of the model. Assessment refers to evaluating the organisation's strengths and weaknesses compared to industry best practices, and improvement enables an organisation to develop an improvement plan to increase its maturity level.
In terms of assessment OPM3® has both a high-level, self assessment module (SAMS) and a more comprehensive assessment. The self-assessment requires answering a simple yes or no to whether 151 high-level best practices exist in the organisation. The resulting output provides a list of best practices the organisation currently demonstrates and those that it does not. The comprehensive assessment is an evaluation of specific supporting “capabilities” that contribute to the best practices and whether they do or do not exist in the organisation. This provides a more comprehensive analysis of the organisation's current state of maturity. The model enables either an internal self-assessment or assessment by an external third party.
OPM3® has a number of distinguishing features, including best practices, capabilities, outcomes, and key performance indicators (KPIs). OPM3® is also a “causal model” that describes causes and effects and identifies and documents observable results. Thus, there are relationships between all of the key elements of the mode—see Exhibit 1. Best practices are defined as elements recognised as representing maturity in organisational project management. Best practices are underpinned by capabilities that aggregate or lead to best practices. There are also dependencies amongst the capabilities and best practices so that certain capabilities must exist in order to achieve a particular best practice, as shown in Exhibit 2.
Exhibit 1 – Relationships Between Key OPM3® Elements
Exhibit 2 - Interdependencies Between Best Practices and Capabilities
The existence of capabilities are evidenced by “outcomes,” which are the tangible or intangible result of applying a capability. Outcomes are quantified by key performance indicators (KPIs) or “criteria by which an organisation can determine whether the capability exists or the degree to which it exists” (PMI, 2003). KPIs describe what a user should look for to determine whether an outcome corresponding to a capability had been produced.
In contrast to other maturity models, such as Software Engineering Institute's (SEI) Capability Maturity Model (CMM), which have a linear approach to maturity, OPM3® provides a multi-dimensional view of maturity across three “domains” of project, program, and portfolio management. A link to the PMBOK® Guide was incorporated into the model, because users wanted a direct relationship between it and OPM3®. The model also incorporates a process model. Although this adds more complexity to the model, it recognises the importance of underlying processes and process improvement to the discipline of project management. Four stages of process improvement were incorporated: standardise, measure, control, and continuously improve. OPM3® also includes the five lifecycle project management processes of initiating, planning, executing, controlling, and closing, as this was the means of linking OPM3® and the PMBOK® Guide. Because organisational project management requires an understanding of portfolio and program management as well as project management, the five PMBOK® Guide process groups were extended to the portfolio and program management domains.
Also added to the model were additional features called “organisational enablers,” which must exist to enhance the benefits of best practices achieved by the organisation. They are considered a special set of best practices, and examples include executive sponsorship, organisational project management policy and vision, and teamwork approaches. OPM3® provides data that enables organisations to assess their organisational project management maturity along a continuum across the different domains of portfolio, program, and project management and the five process improvement stages.
OPM3® Cases
A number of OPM3® cases from Australia, New Zealand, and the United States were examined to determine the benefits of undertaking maturity assessments. Some of this data was publicly available, other data was proprietary. This section will examine these cases in detail and provide details on the organisational context, the type of assessment undertaken, the scope of the assessment, and the triggers for the assessment, as well as the data sources for the assessment, the assessment outcomes, and the benefits from the assessment.
Centrelink
For organisational context, Centrelink is the primary social security agency for the Australian Government and ranks in the top 100 companies in Australia by size and turnover (Centrelink, 2006–2007). An indication of the scale and scope of the organisation is provided by the following statistics: 6.5 million customers (about a third of the Australian population), pays 10 million individual entitlements annually and records 5.2 billion electronic transactions annually, administers more than 140 different products and services for 25 government agencies, employs more than 25,000 staff, has more than 1000 service delivery points, and& receives more than 31 million telephone calls annually.
The assessment was a performance audit (Centrelink, 2007) carried out between February and October 2006 by the Australian National Audit Office (ANAO). The ANAO undertakes performance audits and financial statement audits of Commonwealth public sector bodies, and gives independent reports and advice to improve Commonwealth public sector administration and accountability. The ANAO was assisted in the audit by an external consulting firm.
The purpose of the audit was to assess the effectiveness of the major elements of Centrelink's central, strategic level project management arrangements. The scope of the audit included review of key documentation, observation of project management practices and confirmation of findings with key staff, analysis of a sample of projects for adherence to the corporate project management framework, and a more detailed review of four projects to assess strategic project management arrangements.
The source of data was the ANAO report, which is publicly available. It provides a rich source of project management-related data and, in contrast to the other OPM3® cases, represents a more independent, objective review of project management maturity. It also provides a unique insight into the constraints the project management discipline operates under in the government sector.
During the ANAO audit Centrelink undertook an OPM3® self-assessment, although the audit report indicates it was a limited exercise based on self-assessment surveys completed by internal stakeholders. It is not known why this assessment was done, although perhaps an external assessment triggered a desire for the organisation to undertake their own internal assessment. An interesting development was that the ANAO also undertook a maturity assessment using the P3M3 model (Portfolio, Programme, and Project Management Maturity Model) released in February 2006 by the U.K. Office of Government Commerce (OGC, 2006). This model is similar to the SEI Capability Maturity Model (CMM) in that it has a matrix of five levels (initial, repeatable, defined, managed, and optimised) against project, program, and portfolio. And like CMM, it has key process areas (such as benefits management). Using two maturity different models is an interesting approach, because the P3M3 finding correlated with the OPM3® results (see Exhibit 3).
Exhibit 3 – Centrelink on OPM3® Continuums
In terms of assessment outcomes and results, the ANAO audit finding was that Centrelink concentrated on the management of individual projects rather than programs. The OPM3® assessment returned an overall maturity rating of 34% (225 out of 586 best practices), with project management at 50%, program management at 39%, and portfolio management at 18%. In terms of process improvement stages, 49% of the best practices were in the standardise stage, 22% in the measure stage, 36% in the control stage, and 21% in continuous improvement. P3M3 results indicated that Centrelink had most processes in place for levels 1 and 2 (that is, initial and repeatable).
The external audit conducted by ANAO provided the organisation a number of benefits, including identifying areas for improvement, including improved definition of projects and programs of work, the improved use of program-level monitoring and reporting, improving the use and integration of IT tools to support project managers, the central project office working more closely with project sponsors and managers, and more effective evaluation of projects at completion. The audit also noted that the organisation would benefit from moving towards establishing defined processes in line with P3M3.
The ANAO audit also enabled a review of previous audits or assessments by ANAO to determine how they were progressing. For example, a previous ANAO audit in response to a Centrelink project had introduced mandatory qualifications for all project managers. A review indicated that there had been a large degree of improvement. The ANAO audit also noted that organisational plans for project manager training, status, support, and career pathways were positive steps towards increasing strategic project management maturity.
The audit also provided some internal and external benchmarking data. For example, the ANAO recognised, by comparing Centrelink current practices with a KPMG Project Management Survey (KPMG, 2005), that Centrelink's initiatives and approach are ahead in some areas compared to others (Centrelink, 2007, p. 98). The ANAO audit report also highlighted elements of internal best practice. For example, certain organisation units were recognised internally as being leaders in scheduling practice, providing opportunities to transfer these best practices to other areas of the organisation.
The audit highlighted the extent of organisational commitment to improving project management. This commitment was documented from the organisation's senior executive and acknowledged from the auditor in the report. The ANAO noted that Centrelink implemented a project management framework based on PMBOK® Guide and PRINCE2 and has been “… continuously improving it since then. The ANOA recognises Centrelink's organisational commitment to continue with this process” (Centrelink, 2007, p. 98).
New Zealand Public Sector Organisation
In 2003 an OPM3® assessment (SIMPL, 2007) was carried out on a public sector organisation by a third party, SIMPL, an IT and consulting company, both based in New Zealand. The name of the client agency has been with withheld by the request of SIMPL. Like the previous OPM3® case, the client organisation is a public sector firm, although relatively smaller. From New Zealand's perspective, however, it has a large capital budget, is project-orientated, and has many manual processes. It has approximately 700 staff, primarily employees, with about 50% working on projects at any one time.
The assessment undertaken was a high-level, OPM3® self-assessment led by a SIMPL team leader, with 8 people from the client organisation. The assessment was done in a series of workshops, with follow-up where there were questions or areas requiring further clarification or interpretation. Unlike the previous OPM3® case, which was an external assessment performed by an independent third party, the trigger for the assessment in this case was that SIMPL had been engaged to do a requirements-gathering exercise and business case for the purchase of an enterprise project management tool. The client was interested in seeing how it measured up currently now versus the proposed state, with the tool being used as a means of improving their project management maturity. Unlike the previous OPM3® case, where the data is publicly available, the data for the New Zealand case was based on interviews and follow-up by this paper's authors with the SIMPL team leader responsible for the assessment.
As shown in Exhibit 4, the outcomes from the OPM3® assessment indicated that the organisation had an overall maturity rating of 41%, with 56% of best practices in project management, 43% in program management, and 28% in portfolio management. With process improvement, 37% best practices were in the standardise stage, 35% in the measure stage, 24% in the control stage, and 69% in the improvement stage (this last score seems odd given the large number of manual processes).
Exhibit 4 – New Zealand Public Sector Organisation on OPM3® Continuums
In terms of challenges, the organisation had a large number of manual processes, which would have made it difficult to standardise, measure, and improve its processes. The organisation was also implementing a large number of initiatives simultaneously, including selecting a project management enterprise tool, documenting business processes, establishing a project management office, and assessing project manager competency using a hybrid of PMI's Project Manager Competency Framework and New Zealand government requirements. It should be noted that this was also very early days for OPM3®, and there weren't many test cases that could be referred to. The SIMPL team leader also commented that in using OPM3® he had to do a number of iterations of the report, as the client questioned the scores and this required him to use his judgement to interpret the OPM3® output. He also commented that even though OPM3® applied to any size organisation, some of the best practices were not relevant or may not provide additional value given the relatively small size of the organisation.
Even though this was an early example of an OPM3® assessment, it provided the organisation a number of benefits. For the client organisation, the assessment identified “quick wins”—best practices that it currently did not have that could be attained relatively quickly. The assessment also helped kick off other related initiatives, such as project management training and selecting an enterprise project management tool. The assessment also helped the client to define more clearly what it wanted to achieve. Importantly the assessment provided a baseline or stake in the ground against which the company could measure itself: for example, in the future it could see the impact of the various initiatives it had rolled out. Interestingly, the assessment helped articulate the potential benefits of the enterprise project management tool, in particular which best practices the tool could help achieve. The assessment also helped add weight to the business case for the overall change program. For the consulting organisation the assessment was an opportunity to test their understanding of OPM3® and use a maturity model in an actual situation.
Pinellas County IT
Pinellas County IT (Pinellas, 2006) was the first published OPM3® PMI case study. The organisation is the IT department that supports the government of Pinellas County, Florida, U.S.A., which has 24 municipalities and 930,000 people. It provides services to other government departments; however, these agencies can choose between using the Pinellas IT department or engaging external contractors. During 2003, government departments increasingly began to choose external providers for their IT services because Pinellas had gained a reputation for failing to deliver projects on time and within budget. Around this time the department's senior executive determined a new vision for the department: to become the service provider of choice.
A first high-level self-assessment in 2004 involving 2 people revealed an overall maturity rating of 42% and a large number of best practices that needed improvement (see Exhibit 5). The organisation then selected five areas of priority—initiation, scheduling, resource management, closure, and execution—based on urgency, alignment with strategic objectives, and so forth. They developed an improvement plan and had a second assessment in late 2004 that involved a larger number of participants and an increased score of 48%, indicating improvements were working. A third assessment in 2005 had even more participants (22) with a third from customer departments. This time the overall maturity rating decreased to 46% although this was put down to people becoming more critical and demanding more performance. A fourth assessment in 2006 indicated further improvement, with the overall maturity score increasing to 49%.
Exhibit 5 – Overall Maturity Rating for Pinellas County IT
The benefits that Pinellas County IT obtained from the assessments and the improvement plans that it implemented included being able to associate specific assessment data with certain groups of participants, which meant it could identify key issues or accomplishments and respond to these. This was a somewhat unexpected benefit. Undergoing the OPM3® assessment and improvement initiatives also provided additional motivation for people working in projects, as they could see where they, as individuals, fitted into the organisation. The process also helped turn around customers, who are now more likely to consider the department for IT work. The organisation reported that more projects were being completed, evidence of improved project management maturity. However the case study also highlighted the difficulties of being able to measure benefits in terms of cost savings resulting from initiatives—in maturity models this is assumed to be relatively straightforward.
This case study also provided some important practical lessons learned for those undertaking assessments and improvements. This included the importance of getting senior management support for and commitment to the initiative. The case also showed the importance of managing scope—don't try to bite off too much too soon. Instead, focus on selected important areas for improvement and proceed in small steps. The goals or objectives should also be realistic and attainable. The case study also highlighted the importance of managing the cultural change that arises, and the importance of regular communication.
Savannah River Site
The Washington Savannah River Company (WRSC, 2006) was the first assessment of the OPM3® Product Suite, which comprises software, training, and certifications of individuals as OPM3® assessors and consultants. The assessment team of four volunteers conducted the first OPM3® Product Suite Assessment for WSRC during 27–31 March 2006. Team members interviewed more than 20 site staff members to look for evidence that they were achieving outcomes expected from the best practices. They also confirmed whether answers were accurate by checking project documents, such as project schedules. The main objectives of the assessment were (1) to conduct an organisational project maturity assessment using OPM3® Product Suite methodology and tools in an industry application and (2) evaluate the effectiveness of the OPM3® Product Suite methodology and tools.
WSRC is a subsidiary of Washington Group International. With headquarters in Boise, Idaho, it has revenues of $3 billion $US and approx. 24,000 employees globally in heavy industries (such as defence, power, and environmental cleanup), with the company mainly involved in environmental cleanup projects. The project management culture was highly developed within the company. The scope of the project conducted in 2006 included the four process improvement groups—standardise, measure, control, and improve, and two out of the three domains—project and program management. Portfolio management was excluded from the assessment because the parent company made decisions on portfolio management.
WSRC's assessment results were very high, with organisational enablers (executive sponsorship, benchmarking, and so forth) ranked at 97% (see Exhibit 6). These high scores were also reflected in the domain scores: 97% in project management and 94% in program management, with portfolio management not included. Despite WSRC being mature in terms of organisational project management maturity, the assessment still yielded some benefits. This included validation that it was managing projects well and effectively adapting project management principles. The assessment also provided WSRC with knowledge of specific areas (some best practices were not fully implemented in project team development, program team development, program initiation, and so forth) that could be improved as part of an internal project improvement process.
Exhibit 6 – Overall Maturity for WSRC
Discussion and Conclusion
The cases analysed in this paper have demonstrated the benefits and issues in project management maturity assessments. This section will discuss these by comparing and contrasting the OPM3® cases. In Centrelink's case, the benefits from three types of assessments (OPM3, P3M3, and the audit) included highlighting improvement opportunities in project definition, program management, defining processes, and project closure. This case also highlighted another benefit of maturity assessments—results from previous improvement initiatives (such as project manager training) could be reviewed. The audit also indicated some internal best practices (such as in scheduling) and that it could be benchmarked favourably to its peers in some areas of project management. However, for Centrelink to capitalise on these assessments it would need to transfer internal best practices, determine which opportunities to implement, and do more maturity assessments to see their impact.
The NZ case in contrast was different. There were no previous assessments or improvement initiatives that could be reviewed. A single OPM3® assessment was done under the direction of an external consulting firm and a team of client representatives, rather than internally only. The assessment data, rather than being publicly available, was restricted to interviews with the SIMPL team leader responsible for leading the assessment. Despite this, the assessment provided real benefits for both the client and the consulting firm (although qualitatively different). These included support for current client initiatives, identifying best practices within reach, and establishing a performance baseline. For the client it provided valuable experience in understanding and using OPM3® in real life. However, this case also pointed out that OPM3® is complex and requires judgement and interpretation of results. As with the Centrelink case, more assessments are required to determine the impact of its various initiatives
The Pinellas County IT case was was different from the previous cases, because it provided a series of assessments and resulting improvement plans that could be reviewed over a period of 2 years. The benefit here was that Pinellas IT could see improvements in its maturity from implementing initiatives. Other benefits were more qualitative, and included improved employee morale and increased likelihood that customers would use Pinellas IT. The case also provided practical lessons, including having realistic goals, getting executive buy-in, and that cultural change and communication need to be managed. It also indicated the practical difficulties of measuring the impact of improvements made.
The Savannah River Site case was unique, because the organisation was highly developed in its project maturity and the assessment involved using the OPM3® Product Suite components. However, the case showed that, even for organisations that have high project management maturity, areas for improvement could be identified.
These four OPM3® cases are summarized in Exhibit 7.
Exhibit 7 – Summary of OPM3® Cases
The OPM3® project management maturity model, despite being released in 2003, is still relatively new to the marketplace and to practitioners. There are also few published OPM3® cases currently available to review and analyse. The OPM3® Product Suite has been released; however, there are only a few accredited OPM3® Assessors and Consultants in Australia and New Zealand, although there are more in the United States. It would be useful to undertake further analysis to understand the opportunities and constraints to more widespread use of maturity models like OPM3® and the benefits it would provide in additional case studies. However, the OPM3® cases analysed in this paper show there are organisational benefits for organisations to perform maturity assessments, although this should be tempered with some of the practical challenges and issues in using more complex tools.