Benefits management in practice
an exploratory investigation
Les Labuschagne, School of Computing, University of South Africa, South Africa
Most programs are undertaken to deliver benefits to their stakeholders. The discipline of program management attempts to maximize this value by managing several projects collectively to culminate into a final product. PMI's Standard for Program Management represents the consensus of practitioners as to what constitutes best practice, defined as “a technique or methodology that, through experience and research, has reliably led to a desired or optimum result.” According to this definition, experience can be interpreted as the actual application of a technique or methodology. Part of program management involves the management of program benefits. This suggests that most organizations are practicing formal program benefits management to plan and track the benefits from identification to transition. However, anecdotal evidence suggests that this might not be the case in most South African professional services organizations, and this paper reports on exploratory research that was initiated to establish the current practice of program benefits management as part of the wider field of program management. This study is based on five hypotheses, the investigating of which involved semi-structured interviews to gather data that would support or debunk the anecdotal evidence. This paper reports on the research process and results achieved, and therefore it represents research in progress. By developing an understanding of trends and practices in large, professional services organizations, specific challenges can be identified that can be further researched in order to contribute to the evolution of best practices.
Most programs consist of multiple projects and are undertaken to deliver a product or service that has value or benefit to their stakeholders. The discipline of program management (Artto, Martinsuo, Gemünden, & Murtoaro, 2009) attempts to maximize this value by managing several projects collectively to achieve cumulative benefits. Program management as a discipline is evolving, as is evident from the release of the second edition of the Standard for Program Management by the Project Management Institute (2008). “A program is a group of related projects managed in a coordinated way to obtain benefits and control not available from managing them individually” (PMI, 2008). Program management is defined as “the centralized coordinated management of a program to achieve the program's benefits and objectives.” Benefit emerges as a central theme for both of these definitions, and benefits management is therefore central to the practice of program management.
Despite the existence of a standard (PMI, 2008) and best practices (Artto et al., 2009; Pellegrinelli, Partington, Hemingway, Mohdzain & Shah, 2007) in literature, anecdotal evidence, gathered from discussions with project and program managers, suggests that program benefits management is not widely practiced in professional service organizations in South Africa. For the purposes of this paper, professional services organizations are defined as those organizations that render non-tangible, value-adding services to a customer base such as banks, telecommunications and insurance. Although no literature was found to substantiate this claim, the discussions revealed that program benefits management is either not done at all or done in a haphazard manner. Therefore, it can be interpreted that there might be a gap between what literature suggests should happen in practice and what practitioners perceive to be happening in reality. In order to develop a better understanding of this perceived gap, scientifically gathered data is required. This research initiative therefore attempts to establish current practices in program benefits management within the professional services organizations domain in the Republic of South Africa (RSA). According to the official national website, RSA is the economic powerhouse of Africa, with its banking sector rated among the top 10 globally (Brand South Africa, n.d.). The country ranks 26th out of 240, based on population size (Wikipedia, n.d.).
The goal of this paper is to establish whether program benefits management, as presented in literature, is being practiced. The first objective is to explain the research process that was followed to gather factual data. The second objective is to report on the analysis of the data that was obtained. The paper begins with an explanation of the research questions that were developed, followed by the research methodology that was used to investigate them. A brief overview is given of the current literature on the topic, followed by an interpretation of the results.
Best practice is defined as “a technique or methodology that, through experience and research, has reliably led to a desired or optimum result” (Jarrar & Zairi, 2000). According to this definition, experience refers to the actual application of a technique or methodology and it can therefore be deduced that the Standard for Program Management (PMI, 2008) may be considered as a best practice as it represents a consensus of persons with an interest in the topic. Furthermore, PMI® global standards are developed using a three-step process for review and approval steps:
- Each standard is created or updated by a global group of volunteers currently working in the specific field.
- Standards Program Working Sessions enable the volunteer teams to receive feedback from professionals in the field.
- Prior to publication, standards are made available to the general public for input through an exposure draft process.
Stated differently, several persons have agreed that the practices contained in the standard have actually been used with success. As program benefits management is part of program management, it can also be deduced that program benefits management is practiced in most organizations.
Using these deductions as the foundation for the research, five research questions were developed.
- RQ1 – Do strategic objectives guide the conceptualization and selection of initiatives and/or projects?
- RQ 2 – Are benefits determined and quantified before the start of all projects or programs?
- RQ 3 – Are the benefits associated with a project or program major determinants in its selection and funding?
- RQ 4 – Are benefits tracked and measured after completion of projects or programs?
- RQ 5 – Is project success measured against the benefits that were realized?
Next is an explanation of the research questions, together with a summary of the findings from the literature review.
Research Question 1 – Do strategic objectives guide the conceptualization and selection of initiatives and/or projects?
Projects and programs are chosen based on their contribution to the overall organizational need. The strategic objectives guide the organization in its activities while initiatives that do not contribute to the achievement of the strategic objectives are not supported.
Strategic planning is a well-documented management discipline and entails planning for the future (Gupta, Boyd & Sussman, 2004). The primary responsibility of a board of directors is to outline a vision and then determine organizational strategies for achieving the changes necessary to fulfill it within a predetermined time frame (Lint & Pennings, 1999; Spanner, Nuňo & Chandra, 1993).
The need to align project delivery capability with corporate strategy is well recognized. Organizations undertake a multitude of projects to pursue their specific initiatives and projects (Cooke-Davies, Crawford & Lechler, 2009), and provide the general structures, standards, and regulations in an attempt to ensure satisfactory governance and successful accomplishment of these projects. The organization's strategy has consequences not only for the choice and funding of a specific project and the definition of its goals, but also for what is valued and the way outcomes are achieved and reported. Cooke-Davies et al. (2009) concluded that it is the organization's strategy that drives its selection of projects.
This train of thought has been raised earlier by other authors, who stated that project management had been recognized as a strategic delivery capability (Besner & Hobbs, 2006; Crawford, 2004) that could assist organizations in achieving their strategic objectives (Blichfeldt & Eskerod, 2008; Crawford, Hobbs & Turner, 2007). Indeed, organizations are increasingly realizing that corporate strategy is delivered through projects and so project management capability is key to their ability to deliver their strategic intent (Crawford et al., 2007).
Milosevic and Srivannaboon (2006) saw the essence of project management is as being to support the execution of an organization's competitive strategy in delivering a desired outcome. Despite this recognition that projects need to be derived from the strategic objectives, few organizations have a structured, documented process for deriving projects from the organizational vision, but rather opt for a more intuitive approach. Portfolio management is a relatively new management discipline that attempts to bridge the gap between vision and projects (De Reyck, Grushka-Cockayne, Lockett, Calderini, Moura & Sloper, 2005; Thiry & Deguire, 2007). Several sources suggest different frameworks or models (Comprehensive Consulting Solutions, 2001; Phillips, 2002; Szymczak & Walker, 2003; Walls, 2004), yet no published scientific proof or validation of their effectiveness could be found.
With the advent of corporate governance (Thiry & Deguire, 2007; Turner, 2006), a global economy that is going through a recession (Yandle, 2009) and a severe skills shortage worldwide (Morello, 2008), developing better strategic delivery capability and getting value out of all investments have become imperative.
Research Question 2 – Are benefits determined and quantified before the start of all projects or programs?
Feasibility studies are performed before the initiation of any project or program. The feasibility study includes the financial feasibility and envisaged benefits.
Benefits should be identified and quantified before the project begins (Remenyi, Money & Bannister, 2007). For a project to be judged a success, potential benefits need to be identified as early as possible (Remenyi, 1998). This constitutes the first step of the benefits management process, identifying and documenting benefits that will be most relevant and convincing to decision-makers (Bennington, 2004). In general, it can be stated that the proposed benefits from a project must link in some way to the objectives of the organization itself (Dhillon, 2005).
The Standard for Program Management states that the identification of benefits starts during the preprogram preparations (PMI, 2008). This implies that the benefits must be identified before initiation of the projects that form part of the program. Williams and Parr (2008) agreed that the process of benefits management begins with the identification of benefits before a program is initiated, and continues with the measurement of the benefits even after the program has been delivered. It can therefore be concluded that in both program and project management, the benefits must be defined before the program and project are initiated. The implication is that it is a function within both disciplines and is not mutually exclusive to either.
Research Question 3 – Are the benefits associated with a project or program major determinants in its selection and funding?
All projects and programs begin with an investigation into the benefits that will be derived. Various potential projects are compared with one another to establish the optimum program composition and benefit that can be realized. Projects and programs that cannot show clear benefits are not approved. The selection of projects is difficult because there are various quantitative and qualitative factors to be considered, such as organizational goals, benefits, project risks, and available resources (Chen & Cheng, 2009). The main reason that benefits are identified and quantified seems to be a need to gain project approval (Dhillon, 2005). This implies that well-structured project selection criteria will help ensure that organizations select projects that will best support organizational needs. It further identifies and analyses risks and proposed benefits before funds and resources are allocated (Stewart, 2008). However, often benefits are primarily strategic or tactical in nature and their financial rewards are difficult to forecast. The purpose of a formal project selection process is to weigh the risk and rewards of each identified project, based on whether it is ideal and pragmatic to initiate (Williams & Parr, 2008).
The selection of projects based on their respective benefits is directly linked to hypothesis 1, which states that strategic objectives guide the conceptualization and selection of initiatives and/or projects. The ultimate benefit for any organization is the realization of its vision and strategies.
Research Question 4 – Are benefits tracked and measured after the completion of projects or programs?
Once a project or program is completed there is a concerted effort to determine the amount of benefits that was derived from it in comparison to what was forecasted. This is measured on a regular basis throughout the lifespan of the result of the project or program.
For Bennington (2004), benefits realization is the comparison of the actual delivered benefits against the planned benefits of a project, and the measurement of the planned benefits is supposed to take place at the beginning of the utilization of the resulting product, as well as when it has been in operation for a while. Williams and Parr (2008) added that the review of benefits may be carried out at any suitable time during the execution of a program, whereas for PMI (2008), benefits realization spans a wide variety of factors that include strategic alignment of the project, the value delivered by the project, and the management of resources and risk.
As early as 2003, Lin and Pervan noticed that CIOs were often under immense pressure to find ways to measure the contribution of IT investments to business performance (De Reyck et al., 2005). CIOs also struggled to find reliable ways to ensure that the business benefits from the investments were actually realized. Dhillon (2005) confirmed this as figures coming from the United States of America (USA) suggest that nearly $59 million were being spent on project cost overruns, and some $81 million in cancelled IT projects.
Ward and Daniel (2008) underlined the above claims through research that indicates only 26% of organisations review projects after completion to determine whether benefits were delivered.
Research Question 5 – Is project success measured against the benefits that were realized?
Organizations are moving toward measuring the success of the project, based on benefits realized, as opposed to measuring the success in managing it. The benefits realized are also compared with the original business case that attempted to forecast them.
A project's success is achieved when it is perceived to be such by the stakeholders (Thomas & Fernández, 2008). Project success therefore extends beyond technical performance, cost, and duration to dimensions such as user satisfaction and the benefits that were identified at its onset. Cooke-Davies (2002) distinguished between project management success, which is measured by time, cost and quality, and project success, which is measured against the overall objectives of the project. For Dhillon (2005), a well-defined benefits management process is the most significant factor in the success of IT implementation.
By investigating the discussed research questions, a better understanding can be developed of the current practices around program benefits management. Given the prevailing global economic climate, it is assumed that organizations are scrutinizing all major expenditures and investments in order to ensure optimal utilization of funds.
The next section elaborates on the research methodology used to investigate the research questions.
A qualitative research methodology was used to seek illumination and understanding through extrapolation of the findings at hand. This methodology was selected in favor of a quantitative one as the researchers wished to gain an in-depth understanding of the practice within the context of the standard. A quantitative research methodology would have been limiting as it focuses mainly on the relationships between various sets of facts (Creswell, 2003; Bell, 2007, p. 7).
Within the qualitative research methodology, various research methods exist, such as interviews, case studies, observation, and action research (Altrichter, Kemmis, Mctaggart & Zuber-Skerrit, 2002). Interviews were chosen as the research method as they allow the researcher to understand more fully the subjects' experiences, as well as to learn more about their answers to the questions posed (Cunningham, 2008). The advantages of the interview research method are that it provides a broad range and depth of information, as researchers develop a relationship with the subjects and they can be flexible during the interview itself (Kwok & Ku, 2008).
The following process was applied by the researchers to gather the necessary information: first, an extensive literature survey was conducted to determine the best practices involved in program management and program benefits management. Twenty-one references were consulted in relation to program management, ranging between 2001 and 2009. Five references were made to books and 16 to peer-reviewed journal articles. For the purposes of this paper, the Standard for Program Management was classified as a book. Some 34 articles were referenced regarding benefits management, dating between 1998 and 2009, and consisting of two books and 32 peer-reviewed journal articles. This literature review provided the researchers with the necessary theoretical foundation that was needed to proceed to the next step in the research process.
The second step in the process was to develop a semi-structured interview guide focusing on three aspects:
- The interviewee's role and responsibilities within the organization.
- The purpose of exploring this aspect was to determine if the interviewee's job description was in line with the literature, and so to ensure that the appropriate interviewee had been selected. It also determined whether the interviewee responded to the various questions from an authoritative, participative, or awareness perspective, based on the length of employment within the position itself, as well as within the organization.
- The organizational strategies that were initiated and executed, the role of business cases, as well as the way projects were initiated and linked to the organizational strategies.
- This aspect is the essence of the research and the various questions within this section were based on the findings of the literature survey. The aim was to determine if practice follows the theoretical best practices as suggested by literature.
- The summation of the business strategies of the organization as well as the perception of the interviewee regarding organizational success.
- This aspect focused on the interviewee's perception of organizational success. Perception was based on personal belief rather than on factual evidence. It also focused on additional information that the interviewee felt would provide context to the interview.
The third step in the research process was to identify program managers, portfolio managers, functional unit managers, and C-level executives of organizations who could participate in interviews. Program benefits management is a function of program management and is therefore managed by program managers (PMI, 2008), or in some cases by portfolio managers. The inclusion of functional unit managers and C-level executives was prompted by the fact that some organizations do not have program managers per se. In these organizations the responsibility of program management is often allocated to a functional unit manager or a C-level executive.
The interviewees were identified through two processes:
- Targeting large professional service organizations and requesting a list of program managers and portfolio managers employed by them. The identified managers were then contacted directly and invited to participate in the research. The main criterion that was used to identify potential organizations was that they had to be listed on the Johannesburg Stock Exchange (JSE). The rationale was that these organizations were more likely to adhere to best practices and industry standards, and that they would use projects as a vehicle for the implementation of strategic objectives.
- Collaborating with Project Management South Africa (PMSA), a professional body for project, program, and portfolio managers in RSA, to identify members who had program management responsibilities. PMSA then invited these individuals on behalf of the researchers to participate in the research.
In both cases, participation was voluntary and formal permission was obtained from participants to use the results of the interviews for the study. Assurance was given that all results would be treated as confidential and that anonymity would be ensured. All interviews were conducted by a minimum of two researchers, with Researcher One conducting the interview while Researcher Two recorded the interview using a digital voice recorder and took notes. The interviews were then transcribed directly from the digital voice recordings and the transcripts checked by the researchers for accuracy and correctness by comparing them to the digital voice recordings. The transcripts were then sent back to the interviewees to verify that they were an accurate and authentic copy of what was said in the interview. Interviewees were given the opportunity to change or remove anything with which they did not feel comfortable. The verified transcriptions were rendered anonymous then loaded into a computer-assisted qualitative data analysis (CAQDAS) software package (ATLAS.ti version 6) for analysis, along with any supporting documentation (Lewins & Silver, 2008). Most of the CAQDAS packages provide the same functionality and usage is based on the personal preferences of the researchers (Rettie, Robinson, Radke, & Ye, 2008).
The package enables the researchers to “code” the transcriptions for analysis purposes; that is to test the relationship between issues, concepts, and themes, and to develop broader or higher order categories (Lewins & Silver, 2008). Coding also facilitates the development of a detailed understanding of the phenomena that the data is seen to be presenting (Atherton & Elsmore, 2007). Coding is influenced by various factors; for example, the research aims and the kind of data, as well as the depth of the analysis (Lewins & Silver, 2008). Codes can be generated inductively or deductively (Mangan, Lalwani, & Gardner, 2004). Inductive codes imply that salient aspects are defined within the data and deductive codes are assigned to predefined areas of interest. The researchers used inductive codes to prevent bias toward any predefined areas of interest. These predefined areas of interest are normally guided by a literature survey (Atherton & Elsmore, 2007; Mangan et al., 2004), which in this study had already guided the composition of the questionnaire. Inductive coding also provided a richness that was not possible through deductive coding.
Inductive coding follows a three-step process (Lewins & Silver, 2008; Von Seggen & Young, 2003):
- Perform open coding—Small segments of data are considered in detail and compared with one another. This step generates large volumes of codes that encapsulate the notion of “what is going on.”
- Perform axial coding—All the codes that were generated are analyzed. Codes are rethought in terms of similarity and difference and consolidated where appropriate.
- Perform selective coding—The researchers revisit the codes and instances in the data that pertinently illustrate themes and concepts are identified. Conclusions are validated by illustrating instances represented by and grounded in the data.
Based on the inductive coding process, the data was analyzed in order to develop a better understanding.
The mapping of the five research questions against the semi-structured interview guide is illustrated in Table 1. The interview guide that was used was derived from the literature review.
Table 1: Mapping of Research Questions to Semi-Structured Interview Guide
|No.||Research Question||Related Question in Semi-Structured Interview|
|1||Do strategic objectives guide the conceptualization and selection of initiatives and/or projects?|| |
|2||Are benefits determined and quantified before the start of all projects?|| |
|3||Are the benefits associated with a project or program major determinants in its selection and funding?|| |
|4||Are benefits tracked and measured after completion of projects or programs?|| |
|5||Is project success measured against the benefits that were realized?|| |
The next section focuses on the interpretation of the results and provides preliminary explanations of the various research questions previously raised.
Research Question 1 – Do strategic objectives guide the conceptualization and selection of initiatives and/or projects?
The results from the interviews vary as some of the interviewees found it difficult to align projects with strategic objectives, stating that it would always be a challenge in their organizations: “I would like to believe that the projects initiates from a business need. But that's not always the case.”1 This problem arises when organizations do not have sufficient criteria to use for selecting a project or initiative.
Some interviewees stated that their organizations experienced problems despite having a selection and prioritization process in place. Projects driven by a political agenda were also approved, although they might not meet the criteria for project selection. One of the interviewees commented that “It is being very difficult to actually say no to a project.”
Other organizations, however, did not seem to have any problems or issues in deriving or aligning projects and initiatives with strategic objectives. For them, the executive committee determined the overall strategic direction and it was up to the various divisions and units within the organization to determine which projects and initiatives should be implemented to realize these strategic objectives. This phenomenon was found across various types of organizations, ranging from commercial banks to ICT service providers. These organizations also had the necessary criteria in place to select from various projects and initiatives, including budget, technical feasibility, and customer focus: “The strategic plan is EXCO'ed after discussion along with the vote when they ask for a strategic plan. Then from the strategic plan it is shifted down into a functional business plan.”
One aspect that overshadows all the selection criteria is the budget available to the various sections and units. A range of initiatives and projects may have been identified to implement the strategic objectives, but the budget available limits the number that can be authorized. Responses from the interviewees ranged from: “For example, if they say that in the new year they only want to spend a billion rand on projects, then that immediately goes through a re-selection process. Then, of that list they put together, if the list comes to more than a billion [rand] then obviously you've got to take out your lower priority initiative so that would reduce the list of proposed projects for the coming year” to “Based on your business case and based on your priority you get assigned a bucket of money to manage your projects.”
Another aspect that was raised by most of the interviewees was the importance of the business case. The business case provides the linkage between the strategic objectives and the selection of the projects and initiatives. As one of the interviewees put it: “Make sure that there is a business case which can be the source for selecting the right project”. It also became obvious that the onus of writing a business case was the responsibility of the sponsor or owner and not of the project manager and team.
The following conclusions can be drawn from the interviews:
- Organizations that have selection criteria in place that are based on the strategic objectives of the organization use a top-down approach for selecting projects.
- Organizations that do not have selection criteria use a bottom-up approach to link potential projects to the strategic objectives.
The percentage of organizations in which the strategic objectives guide the conceptualization and selection of projects is 81%, implying that 19% or almost one-fifth of the organizations do not follow such a process. This is illustrated in Figure 1.
Figure 1: Percentage of Organizations Where the Strategic Objectives Guide the Conceptualization and Selection of Projects
Research Question 2 – Are benefits determined and quantified before the start of all projects?
Most of the interviewees were in agreement that it is important to determine and quantify the benefits associated with the project or initiative prior to its initiation.
The majority of interviewees stated that the business case was the driver for the quantification of the benefits, making use of net present value (NPV) to quantify the overall benefit of the project. Some of the intangible benefits, quantified over a number of years, are put in monetary value. These benefits include “New business, time savings, just doing things quicker, whatever the benefits may be, and then we also weigh it against the costs.”
An issue with quantifying the benefits at the beginning of a project is “That you're right at the beginning stages and you don't even realize how you're going to be able to do it.” It was also found that organizations matured during the process of quantifying benefits. When they began the process, the quantification was vague, for example: “So our first business cases used to say things like allow the [organization] to do business. You cannot attach a value to that.” As they became more proficient in quantifying the benefits, the metrics associated with the benefit become more rigorous. One interviewee stated: “We did get to the level of really getting to good metrics and the key was also to find out existing metrics.”
One of the interviewees also highlighted the notion of “negative benefits” or penalties. This was particular to several of the major banks in South Africa, which had to become Basel II compliant (Bruggink & Buck, 2002), as enforced by the South African Reserve Bank. Resources had to be utilized by the banks to ensure compliance but, as one interviewee stated, they did not have the resources to become Basel II compliant:
“If the business case is negative that can happen as well with the compliance, SARS [South African Revenue Service] will suddenly say or the Reserve Bank will suddenly say we need a new reporting standard that is negative to the bank. We make that our closing line. That is a negative business case because it now involves technical resources, etcetera, but it has to depreciate as well. It definitely depreciates.”
This implies that a decision is made on cost versus penalties rather than cost versus benefit. In some events the resources required to do such a project would cost more than the penalties incurred for not doing it.
The majority of the interviewees stated that their organizations quantified the benefits of a project upfront. In some instances this was difficult to do, but it was achievable and the quantification was made in monetary terms. This is thus in line with what was found in the literature.
The following conclusions can be drawn from the interviews:
- Organizations use business cases as a mechanism to investigate the feasibility of potential projects. These business cases include the forecasted benefits that will be derived from a project.
- Business cases are often used to make decisions regarding funding of projects.
All the organizations that responded in a positive manner to hypothesis 1 agreed that benefits were determined and quantified before the start of all projects.
The interesting finding is that of the 19% of organizations that responded negatively to hypothesis 1, 33% have benefits that are actually determined and quantified before the start of the project. This implies that although projects are conceptualized and selected using a bottom-up approach, the benefits of such projects are determined and quantified. This is illustrated in Figure 2.
Figure 2: Percentage of Organizations Where the Benefits are Determined and Quantified Before the Start of All Projects
Research Question 3 – Are the benefits associated with a project or program major determinants in its selection and funding?
It became apparent through the interviews that organizations use different ways to select and approve funds for projects. One organization approved all projects, even if they had prioritization in place that used seven criteria to measure benefits. The problem was that the benefit criteria existed but was not being used. “So I would say no, we are really battling to find any criteria to say, at this point, we do not do this project. Although we have the criteria in place now we still find that we approve everything.”
Another organization distinguished between financial and technical feasibility. Projects were funded and selected based on a comparison between the various potential projects, but were, however, approved within the allocated budget. This bottom-up approach was also followed by another organization, in which the selection of the projects was driven largely by the amount of funding it was prepared to spend on initiatives for the year. This implies that in some organizations less important projects are implemented only when there is enough money to fund them: “…then obviously you've got to take out your lower priority initiative so that would reduce the list of proposed projects for the coming year.”
The selection and funding of projects are normally conducted at senior management level within the organization, which might be the executive board, a steering committee, or a governance board. Irrespective of the name of the structure in place, the ultimate decision is made by these committees based on the business case. The response of one interviewee underlines this notion: “If you want a project to run you has to have the business case. You must take it to the ICT Steering Committee.”
Some organizations had a few strategic projects that were initiated by the executive committee, the notion being that the organization needed to implement these projects and initiatives in order to survive. Because of their importance, these projects were funded and became high priorities, and no selection processes or criteria were used for them. These organizations tend to follow a top-down approach, with the strategic objectives being used to derive all the projects. One interviewee underlined this modus operandi when he commented that “… there are 20 projects that get managed at board level.”
The results also indicated that three organizations followed a mixed method to select and fund projects. In some instances, business units selected and funded IT-related projects themselves without notifying the IT division. The implication is that there might be friction between the business units and the IT division as the priorities might be different: “IT has a budget for IT spend as does the business have the approval to do an IT project. It's a mixed bag at the moment and it has been a mixed bag for as long as I have been here.”
The following conclusions can be drawn from the interviews:
- Although the majority of organizations do determine the benefits associated with projects, these benefits are not the always the main reason why projects are initiated.
- The majority of organizations still use a bottom-up approach and projects are driven by the funds available to the organization or business unit rather than what is needed based on the strategic objectives.
- There are, however, a few organizations that use the strategic benefits approach to select the projects that need to be initiated.
Almost half of the organizations (46%) that said benefits are determined and quantified before the start of the project did not use it for project selection and funding. This raises the question of why organizations are going through the motions and exercise of determining and quantifying benefits if they are not to be used in the selection and funding of projects. This is a point of further research that was not addressed in the current research.
Figure 3: Percentage of Organizations Where the Benefits Associated With a Project or Program is Major Determinants in Its Selection and Funding
None of the organizations that form part of the 33% previously mentioned were using it to select or fund projects.
Research Question 4 – Are benefits tracked and measured after completion of projects or programs?
The responses of the interviewees differed significantly but the majority did not track or measure the benefits after completion of the projects. The following extract underlines the general tendency towards benefits tracking:
“There isn't! I had an interesting debate with one of the directors the other day about this point and he believes quite solemnly that there is no justification for benefits realization tracking and his reasons are that by the time the benefits are realized, the organization has changed, the people in the project have either left the organization or they moved to some other area or the complexion of the project changed during its life cycle that the original benefits can't really be tracked. So the argument is that there isn't any real justification for benefits tracking.”
The interviewees were all aware of the necessity to track and measure the benefits and were in the process of putting benefits realization processes in place. These processes were to be used to measure the actual benefits against the envisaged benefits and report on the differences, if any.
A point that was raised during several of the interviews was that the business case needs to be revisited on a regular basis. This is necessary and important because the circumstances when the project was initiated might have changed during the execution of the project and the benefits envisaged might no longer be valid. This leads to a phenomenon where benefits are measured but actually do not make sense because the environment has changed dramatically. This point was raised mainly by the interviewees from the financial sector after the economic turmoil that hit the world in late 2008: “You would have had to revisit the business case anyway; the project is now going in the other direction. So, does the business case during that lifecycle get revisited to see whether it makes sense or not, no I don't think so.”
Benefits also need to be realized in the short term, as is often the case with the implementation of IT infrastructure projects. There is an immediate short-term benefit once the system is in operational mode, but it also provides long-term benefits. In the case of one project mentioned where one of the benefits was to reduce call-waiting, the benefit could be measured in the short term: “During the project the report existed, so it is a case of pulling those reports and you can see the results and you can see the effect of what you were doing during the project.”
From the interviews it can be deduced that projects are more rigorously tracked and measured if the following two aspects are present:
- Budget and funding of project:
- When the project is funded from a strategic pool of resources, then there is focused attention on the realization of the benefits. The same rigor is also present “when an organization's paying sort of five, six hundred million or something over four years they look at the benefits very, very carefully.”
- Strategic value of project:
- Although all projects are supposed to support the strategic objectives of the organization, some projects have more strategic value than others. These projects tend to be scrutinized for benefits realization.
The responses from the interviewees who were C-level executives compared to those who were program managers were very consistent in one respect. Irrespective of whether the interviewee's organization had processes in place to measure and track benefits, the onus of tracking and measurement was the responsibility of the business case owner. The rationale was that if the business case owner determines the benefits upfront, then that person also needs to track and measure the benefits afterward.
The following conclusions can be drawn from the interviews:
- Benefits are generally not tracked and measured after the completion of projects.
- There is an awareness of the need for it but a reluctance to do so.
The organizations that indicated that they are using benefits to select and fund projects are in fact also tracking and measuring the benefits of the project after completion. This is only applicable to organizations that use benefits as a major determinant.
Research Question 5 – Is project success measured against the benefits that were realized?
One of the consistent responses from the interviewees was that there had been a definite shift in the way project success was measured. Where the focus used to be on the triple and quadruple constraints (Besner & Hobbs, 2006; Cooke-Davies, 2002; Thomas & Fernández, 2008), the shift was now more toward the “have we got the benefits of the project that we wanted?” (Yu, Flett, & Bowers, 2005; Patanakul & Milosevic, 2009; Bryde, 2008).
The same phenomenon appeared as with hypothesis 4: project success is measured either immediately after the project is completed or after some time has elapsed. In the first instance, success is determined by how well the project was managed. The second instance relates more to the audit of the product and whether it delivered the benefits envisaged by the business owner and/or organization. This was usually measured in terms of return on investment (ROI) as well as the payback period (PP). Another measure of project success was whether the business owner accepted delivery of the product. One of the interviewees summarized this by saying: “The success of the project from our point of view is on the implementation and acceptance by the owner.”
Some organizations had processes in place to link project success to the benefits. These organizations had the belief that “it relates all the way back to the strategy of the company which will realize whether that was a success or not.” The golden thread is clearly visible where the benefits are derived from or linked to the strategic objectives. The benefits are initially quantified in the business case and upon completion of the project, tracked back to the business case, and then ultimately to the strategic objectives. The project owner/sponsor takes accountability for achieving the benefits: “The project delivers the means by which the benefits are achieved. The business must then take ownership about deriving those benefits.”
There are, however, organizations that did not measure the success of their projects using the benefits achieved. Nor did they track and measure the benefits of the projects. The relationship is distinct: project success cannot be measured in relation to benefits if the benefits have not actually been tracked and measured from the start. One of the interviewees provided the insight needed for this phenomenon: “We made money despite of ourselves and I think it's strangely true of many organizations.”
The following conclusions can be drawn from the interviews:
- There are different interpretations of project success. In some instances it refers to the management of the project, in others, to the ability of the project deliverable to satisfy needs.
- Few organizations link the success of a project back to its strategic objectives. Despite this, they feel that projects do contribute to their strategic intent, even if they are not measured quantitatively.
Summary of Results
Based on the previously discussed findings, the Table 2 is a summary of the comparison between the theory and practice of program management.
Table 2: Summary of Results
|No.||Research Question |
(Theory based on literature survey)
|Finding (Mostly true/Partly true/Mostly false) |
(Practice based on interviews)
|RQ1||Do strategic objectives guide the conceptualization and selection of initiatives and/or projects?||Partly true|
|RQ2||Are benefits determined and quantified before the start of all projects?||Partly true|
|RQ3||Are the benefits associated with a project or program major determinants in its selection and funding?||Mostly true|
|RQ4||Are benefits tracked and measured after completion of projects or programs?||Mostly false|
|RQ5||Is project success measured against the benefits that were realized?||Partly true|
Table 2 shows that program management and in particular benefits management is not yet a widely practiced, formal discipline, but that efforts are being made to change current practices. There is an awareness of what it is and how it should be done, but this has not yet translated into common practice. A major challenge seems to lie in adapting current practices so as to align them with best practices and standards.
It is clear from Figure 4 that only 44% of all the organizations are following the full benefits realization life cycle. Another 43% of organizations are using the benefits of the project to determine selection and funding.
Figure 4: Final Analyses of Data
The negative side is that 19% of all the organizations are not doing any benefits realization.
The paper is based on exploratory research with the purpose of developing an understanding of current practices in the field of program management and, specifically, benefits management. Based on the findings of this investigation, further research will be conducted in the field of program management.
This investigation set out to establish the relationship between the theory and practice of benefits management as part of program management. The study started with five research questions that were developed, based on a comprehensive literature survey. The research questions were then tested by means of semi-structured interviews with practitioners in the field. The preliminary trends were presented and given an indication of the current state of benefits management practices within the South African context.
Based on the preliminary results, it can be seen that there is indeed a gap between theory and practice. Most of the interviewees were familiar with the theory and were in support of its application in their environments. Most also indicated that it was a process and that they had embarked upon it. Despite most interviewees admitting to not practicing benefits management according to the theory, they felt that current practices did contribute to organizational success. The interpretation of organizational success was determined by the interviewee. It could therefore be argued that if current practices are delivering results there is no need to change them.
Some of the factors that were mentioned that prevented the full implementation of benefits management were complexity of the environment, lack of support or understanding from decision makers, immaturity of organizational processes, and inflexibility of current systems, processes, or structures to implement benefits management.
This investigation has shown that there is an awareness of benefits management in most large organizations, and that there are efforts under way to start formalizing its use. This clearly indicates that program management as a practice is slowly following the theory. Despite this, there is still uncertainty within organizations regarding the value of benefits management.
The study further shows that there is a need for benefits management and that organizations can benefit from it. At present, it is still carried out informally in most organizations, due to a lack of formalized knowledge and experience. Program management as a practice is therefore in its infancy and has yet to grow into a generally accepted and proven discipline.
Based on the preliminary results, further analysis will be done on the results to determine the correlation between the different research questions, as well as recommendations made on closing the gap between what is considered best practice and what is currently being done in organizations.
Altrichter, H., Kemmis, S., Mctaggart, R., & Zuber-Skerritt, O. (2002). The concept of action research. The Learning Organisation, 9, 125-131.
Artto, K., Martinsuo, M., Gemünden, H. G., & Murtoaro, J. (2009). Foundations of program management: A bibliometric view. International Journal of Project Management, 27, 1-18.
Atherton, A., & Elsmore, P. (2007). Structuring qualitative enquiry in management and organisation research: A dialogue on the merits of using software for qualitative data analysis. Qualitative Research in Organisations and Management: An International Journal, 2(1), 62-77.
Bell, J. (2007). Doing your research project: A guide for first-time researchers in education, health and social sciences. 4th edition. Berkshire: Open University Press.
Bennington, P. B. (2004). Project benefits management in IT projects—An Australian perspective. Project Management Journal, 35, 20-31.
Besner, C., & Hobbs, B. (2006). The perceived value and potential contribution of project management practices to project success. Project Management Journal, 37(3), 37–49.
Blichfeldt, B. S., & Eskerod, P. (2008). Project portfolio management—There's more to it than what management enacts. International Journal of Project Management, 26(4), 357-365.
Brand South Africa. (n.d.) South Africa: Economic overview. Available at http://www.southafrica.info/business/economy/econoverview.htm. Accessed on 29 November 2009.
Bruggink, B., & Buck, E. (2002). Practical aspects of implementing a Basel II-compliant economic capital framework. Journal of Financial Regulation and Compliance, 10(3), 219-233.
Bryde, D. (2008). Perceptions of the impact of project sponsorship practices on project success. International Journal of Project Management, 26(8), 800-809.
Chen, C.-T., & Cheng, H.-L. (2009). A comprehensive model for selecting information system project under fuzzy environment. International Journal of Project Management, 27, 389-399.
Comprehensive Consulting Solutions. (2001). Project management in an information technology (IT) world (White paper). Brookfield, WI: Comprehensive Consulting Solutions.
Cooke-Davies, T. J. (2002). The ‘real’ success factors in projects. International Journal of Project Management, 20, 185–90.
Cooke-Davies, T. J., Crawford, L. H., & Lechler, T. G. (2009). Project management systems: Moving project management from an operational to a strategic discipline. Project Management Journal, 40, 110-123.
Crawford, L. H. (2004). Global body of project management knowledge and standards. In P. W. G. Morris, & J. K. Pinto (Eds.), The Wiley guide to managing projects. New Jersey: John Wiley & Sons.
Crawford, L., Hobbs, B., & Turner, J. R. (2007). Aligning capability with strategy: Categorizing projects to do the right projects and to do them right. Project Management Journal, 37(2), 38–51.
Creswell, J. W. (2003). Research design: Qualitative, quantitative and mixed method approaches. London: Sage.
Cunningham, W. S. (2008). Voices from the field: Practitioner reactions to collaborative research initiatives. Action Research, 6(4), 373-390.
De Reyck, B., Grushka-Cockayne, Y., Lockett, M., Calderini, S. R., Moura, M., & Sloper, A. (2005). The impact of project portfolio management on information technology projects. International Journal of Project Management, 23, 13.
Dhillon, G. (2005). Gaining benefits from IS/IT implementation: Interpretations from case studies. International Journal of Information Management, 25, 502-515.
Gupta, M., Boyd, L., & Sussman, L. (2004). To better maps: A TOC primer for strategic planning. Business Horizons, 47(2), 15–26.
Jarrar, Y. F., & Zairi, M. (2000). Internal transfer of best practice for performance excellence: A global survey. Benchmarking: An International Journal, 7(4), 239-246.
Kwok, J. Y.-C., & Ku, H.-B. (2008). Making habitable space together with female Chinese immigrants to Hong Kong: An interdisciplinary participatory action research project. Action Research, 6, 261-283.
Lewins, A., & Silver, C. (2008). Using software in qualitative research. London: Sage.
Lint, O., & Pennings, E. (1999). Finance and strategy: Time-to-wait or time-to-market? Long Range Planning, 32(5), 483–493.
Mangan, J., Lalwani, C., & Gardner, B. (2004). Combining quantitative and qualitative methodologies in logistics research. International Journal of Physical Distribution & Logistics Management, 34(7), 565-578.
Milosevic, D. Z., & Srivannaboon, S. (2006). A theoretical framework for aligning project management with business strategy. Project Management Journal, 37(3), 13.
Morello, D. (2008). Focusing on talent in trying times. Available from http://www.gartner.com. (Accessed 5 January 2009).
Patanakul, P., & Milosevic, D. (2009). The effectiveness in managing a group of multiple projects: Factors of influence and measurement criteria. International Journal of Project Management, 27(3), 216-233.
Pellegrinelli, S., Partington, D., Hemingway, C., Mohdzain, Z., & Shah, M. (2007). The importance of context in programme management: An empirical review of programme practices. International Journal of Project Management, 25(1), 41-55.
Phillips, J. J. (2002). The project management scorecard: Measuring the success of project management solutions. New York: Elsevier.
PMI. (2008). Standard for program management. 2nd edition. Newtown Square, PA: Project Management Institute.
Remenyi, D. (1998). Business benefits from information systems through an active benefits realisation programme. International Journal of Project Management, 16, 81-98.
Remenyi, D., Money, A., & Bannister, F. (2007). The effective measurement and management of ICT costs and benefits. Oxford, United Kingdom: CIMA.
Rettie, R., Robinson, H., Radke, A., & Ye, X. (2008). CAQDAS: A supplementary tool for qualitative market research. Qualitative Market Research: An International Journal, 111: 76-88.
Spanner, G.E., Nuňo, J., & Chandra, C. (1993). Time-based strategies—Theory and practice. Long Range Planning, 26(4), 90–101.
Stewart, R. A. (2008). A framework for the life cycle management of information technology projects: ProjectIT. International Journal of Project Management, 26, 203-212.
Szymczak, C. C., & Walker, D. H. T. (2003). Boeing: A case study example of enterprise project management from a learning organisation perspective. The Learning Organization, 10(3), 125-137.
Thiry, M., & Deguire, M. (2007). Recent developments in project-based organisations. International Journal of Project Management, 25(7), 649–658.
Thomas, G., & Fernández, W. (2008). Success in IT projects: A matter of definition? International Journal of Project Management, 26(7), 733-742.
Turner, J. R. (2006). Towards a theory of project management: The nature of the project governance and project management. International Journal of Project Management, 24(2), 93-95.
Von Seggen, M. & Young, N. Y. (2003). The focus group method in libraries: Issues relating to process and data analysis. Reference Services Review, 31(3), 272-284.
Walls, M. R. (2004). Combining decision analysis and portfolio management to improve project selection in the exploration and production firm. Journal of Petroleum Science and Engineering, 44, 55–65.
Ward, J., & Daniel, E. (2008). Benefits management: Delivering value from IS & IT investment. West Sussex, England: John Wiley & Sons.
Wikipedia. (n.d.) List of countries and dependencies by population density. Available from http://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population_density. Accessed on 29 November 2009.
Williams, D., & Parr, T. (2008). Enterprise programme management—Delivering value. New York: Palgrave MacMillan.
Yandle, B. (2009). September 2009-last update, Economic situation report [Homepage of The Strom Thurmond Institute of Government and Public Affairs]. Available: http://www.strom.clemson.edu/teams/ced/esr.html Accessed: September 2009.
Yu, A. G., Flett, P. D., & Bowers, J. A. (2005). Developing a value-centred proposal for assessing project success. International Journal of Project Management, 23(6), 428-436.
1 All responses from interviewees are quoted verbatim, including language errors resulting from English being a second language to some.
© 2010 Project Management Institute