Project benefit management

Abstract

Realizing benefits is an important criterion to evaluate project performance. Hence, project benefit management is essential to enhance project success. This paper focuses on the first step in the benefit management process—the definition and appraisal of target benefits. Target benefits are formulated during project initiation, included in the business case and anticipated to be realized at the completion of the project. Clear definition of target benefits is important for project selection decisions and project portfolio management. This paper develops a new construct—“Quality of Target Benefits” (QTB). Three studies show that a proper formulation and appraisal of a proposed project’s target benefits leads to more informed project selection decisions. The construct can be used to facilitate theory development research on project initiation and performance evaluation and support organizations in making better project funding decisions. The paper contributes to the initiation phase of projects and the project benefit management literature.

Keywords: benefit management; project initiation; target benefits; business case

Introduction

The project management literature is focused on efficient delivery of outputs (artifacts such as a bridge). While this approach is essential, project management methodologies, which focus only on efficient delivery of outputs neglect the importance of project benefit realization. In other words, projects can still be ineffective to the funding organization even if their outputs have been delivered efficiently, for example the Los Angeles Metro (Shenhar and Dvir, 2007) and the Sydney cross-city tunnel (Zwikael & Smyrk, 2011). Moreover, although projects have the potential to enhance organizational performance (Pellegrinelli & Bowman, 1994), project benefits are often not realized (OGC, 2009). Consequently, benefit management is becoming an important research area in project management.

Benefits are first formulated during project initiation in order to be included in a proposed business case. Following the approval of the business case, target benefits become the basis for performance review at the end of the project. For making effective project selection decisions, senior managers should have reliable information regarding the estimated cost, duration and risk level of proposed projects, as well as of its target benefits—the benefits expected to be realized at its completion (Bradley, 2010; Ward and Daniel., 2006). Decision makers rate competing proposed projects against a set of criteria and rank the alternatives according to their overall scores. Hence, the quality of the information decision makers have affects the quality of the decision being made (Mihm, 2010; Raghunathan, 1999). Whereas issues related to quality estimations of project cost, duration, and risk level have been well-studied (e.g., Jørgensen, Halkjelsvik, & Kitchenham, 2012; Vandevoorde & Vanhoucke, 2006; Zeng, An, & Smith, 2007), little is known in the literature about how to ensure the quality of a proposed project’s target benefits. Moreover, target benefits of a proposed project are usually poorly formulated. In particular, Boardman, Mallery, and Vining (1994) highlighted four types of common errors in formulating target benefits – omission, forecasting, measurement, and valuation. Indeed, Lin and Pervan (2003) found that most of the large Australian organizations failed to identify all target benefits of proposed projects. Those identified are often vaguely defined (Norris, 1996), mainly because target benefits are dynamic, uncertain, long term, intangible, and having different meanings to various stakeholders (e.g., Norris, 1996). Although these identified challenges and problems, literature has not provided us sufficient valuable information regarding how target benefits should be formulated and assessed. The little research that has been conducted on this issue is either incomplete or lacks empirical support. For example, Jenner (2009) suggests that target benefits should be ”robust and realizable,” yet it remains vague how to measure these two dimensions and whether they are conclusive.

This paper aims to bridge this gap in the literature by addressing the following research question: “How should target benefits of proposed projects be assessed?”. In order to answer this research question, we aim at developing a measurement tool for a new construct called “Quality of Target Benefits” (QTB) to be defined later in the paper. The developed scale is expected to contribute to both practice and research. In practice, such a scale can be used to guide the formulation of a proposed project’s target benefits and objectively compare target benefits of alternative proposed projects to support informed selection decisions and investment prioritizations. In the remainder of this paper, we first specify the domain of the QTB construct. We then describe our research methodology, followed by the results of three studies. Finally, we discuss the implications and limitations of this research and suggest future research directions.

The Quality of Target Benefits Domain

Churchill (1979) suggested that specifying the domain should be the first step in developing a new construct. Because QTB is a quality-related construct, we first looked at the quality management literature to find that though commonly used, the term “quality” is not consistently defined. Examples of its definition include excellence, value, conformance to specifications, and meeting expectations (Reeves & Bednar, 1994). In this paper, building upon Crosby (1979), we operationalize quality as “conformance to specifications,” as such definition provides an objective basis for the initial assessment of formulated target benefits and allows their monitoring and evaluation. For example, when changes are made to the formulated target benefits during the project life, the quality of the revised target benefits can still be assessed against the same set of specifications. The remainder of this section aims to establish such a set of specifications.

In order to derive potential criteria for assessing quality of target benefits, we draw upon goal-setting theory, because target benefits are strategic goals of a project. Goal-setting theory suggests that goals need to be SMART—Specific, Measureable, Attainable, Relevant, and Time targeted (Doran, 1981). Such SMART goals can enhance performance through directing attention towards goal-relevant activities, energizing effort, encouraging persistence, and leading to strategy development (Boyne & Chen, 2007; Locke & Latham, 2002). The descriptions of the five SMART dimensions and discussions on their relevance to QTB are provided below.

Specific. Goal specificity refers to the extent to which a goal is specifically stated and defined. In the context of target benefits, a vague statement of benefits can lead to an uncertain allocation of resources and responsibility (Norris, 1996). To meet this specificity requirement and prevent different interpretations by various stakeholders, target benefits should have a clear benefit title and description (Zwikael & Smyrk, 2012). They should also be explicitly defined (Breese, 2012; Ward & Daniel, 2006) with a baseline and target value that can be in absolute (e.g., achieve a customer satisfaction score of six on a seven-point scale) or relative terms (e.g., increased current value by 20 per cent).

Measurable. Goals should be measurable to allow organizations to determine whether they have been achieved (Locke & Latham, 2002). In the project context, because target benefits should allow managers to determine whether they have been realized (Ward & Daniel, 2006), they should have agreed measures, clear and relevant units of measurement (Heinrich, 2002; Zwikael & Smyrk, 2012), clear sources of data (Zwikael 7 Smyrk, 2011), and be consistent with those measuring similar project benefits across the.

Attainable. Goals should be challenging yet attainable (Locke & Latham, 2002), to support the commitments and actions undertaken (Maier & Brunstein, 2001). Similarly, target benefits should be realizable (Jenner, 2009); that is, they should be “realistic given the context in which the organization is operating and the constraints it has” (Ward & Daniel, 2006, p. 29).

Relevant. The importance of goal relevance has been emphasized at various levels (e.g., individual, team, organizational), and disciplines (e.g., education, healthcare, and management). For example, Veld and Boselie (2010) suggest that organizations should ensure employees understand the relevance of strategic goals, so they will behave and act in line with these goals. As projects are usually initiated to implement organizational strategies, target benefits should be aligned with, relevant and important to achieving these strategic goals (Ward & Daniel, 2006). For this reason, we suggest that project target benefits should be consistent with an organization’s long-term vision and current strategies.

Time targeted. Goals should have a time frame for completion, which would enable evaluation of their achievement (Locke & Latham, 2002). Such a time frame also plays a critical role in the development of intermediate objectives and strategies for achieving them (Schmidt & DeShon, 2007). Consequently, benefits should have a set target date for their realization (Breese, 2012; Ward & Daniel, 2006) to allow continuous monitoring and tracking as well as the evaluation of their realization.

Research Overview

Our research approach consists of: (1) specifying the domain of the construct, (2) item generation and content validity, (3) initial item reduction. The results of the first step to specify the QTB domain were presented in the previous section, where QTB was identified as a five-dimensional construct and potential items for each dimension were identified. In the following sections, we present three studies which follow the additional scale development steps. To ensure the generalizability of our findings, throughout these studies we recruited participants with relevant experience in formulating and reviewing target benefits of proposed projects from academia and practice from US, China, Australia, Israel, and New Zealand.

In Study 1 we conducted interviews with senior managers, because they could offer insights into the phenomenon under investigation (Churchill, 1979). We used their experience to validate the above-mentioned QTB dimensions derived from the literature, identify missing ones, and generate a final pool of potential measure items. In Study 2 we conducted content validity using a group of international experts from academia and industry. In Study 3, using a survey approach we conducted exploratory factor analysis (EFA) to validate the proposed items and scale structure. The three studies are presented in Table 1.

Table 1: Research Overview.

Study 1 – Item Generation

Participants and Procedure

We conducted a series of interviews with senior managers from eight Australian government agencies. The Australian Public Service (APS) was chosen because of its emphasis on project benefit management practices. For example, benefit realization was highlighted as an important means to assess projects in the 2006 Gateway Review Process (Australian Department of Finance and Deregulation, 2012) and the related Portfolio, Programme, and Project Management Maturity Model (P3M3) and PRojects IN Controlled Environments 2 (PRINCE2) methodology were used by the majority of the Australian government agencies (Gershon, 2008).

Participants were selected through a purposive sampling strategy, based on their involvement and roles in benefit formulation processes. Potential participants were first identified through the Australian government website and invited for interviews via emails and follow-up phone calls. Fifteen participants were chosen based on the relevance of their roles and work experience. Participants were comprised of two directors, one assistant director, one executive secretary, seven national managers, three general managers, and one program manager. The APS classifications of these participants ranged from Executive Level 1 (EL1) to Senior Executive Service (SES) level 3, which are highly senior ranks. The majority of participants were at SES level 1 with an average of 19.2 years work experience and 7.5 years of project experience. All participants had played an active role in benefit formulation or assessment, with experiences ranging from policy development of national importance to the operationalizing of government initiatives. The heterogeneous roles played by different participants allowed us to develop a more comprehensive understanding about practices of assessing the quality of target benefits.

A semi-structured interview approach was chosen to keep the interviewees focused on the interview questions while also allowing them flexibility to discuss additional relevant practices. The interviews were divided into three parts. In the first one, we opened up with “ice-breaking” questions about the participants’ role and responsibilities in formulating target benefits. In the second part, we asked participants to describe how the quality of the target benefits was determined in their agencies. Follow-up questions in response to participants’ answers were then raised to clarify information and explore further details. Sample interview questions included: (1) how do you identify and define target benefits for projects/programs?, (2) how do you determine the quality of target benefits?, (3) what are some factors you think may contribute to the quality of target benefits?, and (4) what do you think can improve the quality of target benefits? Each interview lasted approximately 40 minutes and all interviews were recorded and transcribed for data analysis.

Data Analysis

Content analysis was used with NVivo to study the transcripts through an ongoing iterative process as recommended by Miles and Huberman (1994). First, each transcript was read several times to grasp each participant’s view on the quality of target benefits. Second, all transcripts were re-read to systematically review for consistency and variations in participants’ responses. Then, we amalgamated the interview results and cross-checked them with the findings derived from the literature.

Results

Our interviews confirmed the five dimensions derived from the goal-setting theory by suggesting that QTB can be assessed based on whether benefits fit into organizational strategic goals, have target values and dates, and are measurable and realistic. In addition to confirming the five proposed dimensions, our participants mentioned two additional ones—”accountability” and ”comprehensiveness.” The additional two dimensions are discussed below, whereas illustrative interview quotes for all seven dimensions are presented in Table 2.

Accountability. Our participants emphasized the need to establish clear lines of accountability for target benefit realization for two major reasons. First, accountable managers tend to better ensure that target benefits are properly formulated through consultation with a diverse group of stakeholders, and consideration of the project’s strategic fit. Second, accountability can also help organizations investigating how they may effectively realize the benefits. Assigning a project owner—the person held accountable for securing the project’s target benefits (Zwikael & Smyrk, 2011)—is considered an effective way of addressing accountability in a project context. The literature also agrees that clear and visible line of accountability can further enhance the achievability of target benefits (Breese, 2012; Cooke-Davies, 2007; Lin & Pervan, 2003).

Comprehensiveness. Our participants considered it important to include a complete web of benefits to meet objectives of various stakeholder groups, which is in line with the literature (e.g., Jenner, 2009). Although there is no universal answer as to what can be considered ‘comprehensive’, Henderson and Ruikar (2010) suggest that benefit formulation should rely on diverse categories, including financial/non-financial, direct/indirect, short/long term, internal/inter-organizational, and economical/cultural benefits. Similarly, Irani and Love (2001) suggest that target benefits should comprise of operational, tactical, and strategic natures.

Table 2: Interview Quotes.

Two dimensions were added to those five derived from the literature. 17 proposed items (discussed later) were generated as a result of literature review and interviews to cover these seven dimensions. However, the large number of dimensions and the fact that two dimensions include only a single item do not support an efficient scale. For this reason, consolidation of factors was practiced. Because of their similar nature, the “Measurable” dimension was integrated into “Specific” to create a new dimension called “Specificity”, defined as “the extent to which target benefits are clearly defined and measurable”. “Accountability” and “Time-targeted” dimensions were integrated into “Attainable” to create the second proposed new dimension—“Attainability”, defined as “the extent to which target benefits are realistic, given the context in where the organization is managing the benefit realization process and the constraints it has.” Finally, “Relevant” was integrated into the “Comprehensiveness” dimension, defined as “the extent to which target benefits reflect organizational strategies and the objectives of various stakeholder groups”. This exercise resulted in a three-dimension model. For example, six items are suggested to assess the “Specificity” dimension, including “target benefits were assigned a target value (e.g. 10% decrease in road fatalities),” adopted and modified from Breese (2012), Ward and Daniel (2006), and Zwikael and Smyrk (2012), and “target benefits were assigned measures that are consistent with those measuring similar benefits across the organization,”

Study 2 – Content Validity

The objectives of Study 2 were to: (1) test content validity; and (2) improve clarity, readability and comprehensiveness of proposed items

Participants and Procedure

We used a survey that provided experts with definitions for target benefits and each of the three proposed dimensions. They were then asked to classify the 17 items (presented in random order) into one of the three dimensions where it fitted the best and provide feedback on the clarity and readability of each item (Hinkin, 1998). Experts were not given any indication as to which dimension we expected the items to fall into.

First, a pilot study was conducted, in which one senior academic in the area of management and one research assistant were asked to complete the questionnaire and suggest improvements. Then, for the main study, 24 requests for participation were sent. 21 experts (88%)—16 senior academics with a PhD in management and related areas and five senior practitioners in the area of project management—returned a complete questionnaire. Nine questionnaires were received from Australia, seven from the United States, three from Israel, and two from New Zealand.

Data Analysis

On average, an expert marked the same dimension resulted from Study 1 for 13 items, ranging from 9 to the maximum of 17 correct items. All experts had a correct answer rate within two standard deviations of the mean; hence none was removed from the analysis. The 21 experts also suggested 48 suggestions to improve the clarity and readability of the items.

We used two measures to test the level of agreement of the experts with the proposed dimensions: (1) Substantive agreement—the proportion of respondents who assign an item to its intended dimension. The minimum cut off value suggested for this measure was 0.7; and (2) Substantive validity coefficient—extent to which respondents assign an item to its posited construct than other construct, with a cut off value of 0.5.

Results

Thirteen of the 17 proposed items received higher values than the threshold required for both measures (see Table 3). These items were hence confirmed in their suggested dimensions. Changes were made for items with low scores based on suggestions from experts. Items #11 and #13 were removed due to low scores on both measures and repetition with other items. Item #10 was rephrased and moved from dimension “2” to “1” (Specificity), and items #9 & #14 were rephrased significantly. Finally, based on specific suggestions from our experts, we made change to the wording of other items, as well as definitions of the three dimensions.

*Meets minimum required value

**Dimensions: 1-Specificity; 2-Attainability; 3-Comprehensiveness

Table 3: Content Validity.

Study 3 – Initial Item Reduction

We used quantitative methods to validate the three QTB dimensions and 15 items (see Table 5) proposed at the end of Study 2.

Participants and Procedure

Participants of this study consisted of master of management part-time students at Tsinghua University in China. All students held full-time management positions in various industries, such as finance, manufacturing, service, and engineering. This study involved a cross-sectional survey of senior managers who were engaged in formulating and reviewing target benefits as part of their regular organizational duties. Survey questionnaires were individually distributed and collected by the researchers, including briefing on the questionnaire at the start of the session, and addressing individual questions throughout data collection. To assure equivalence between the questionnaire in Chinese and the original English versions, a standard translation and back-translation procedure was performed (Brislin, 1980): an English version of the questionnaire was carefully translated into Chinese by a bilingual Chinese-English researcher. Then, an additional person who is proficient in both languages was brought in to compare the original questionnaire with the translated questionnaire. This process was repeated until the satisfactory results were reached.

180 questionnaires were distributed in three classes of the same program. Because only participants with relevant job experience in formulating and reviewing target benefits were asked to complete the questionnaire, 48 questionnaires were dropped, leaving 132 complete questionnaires for analysis responses with a response rate of 73%. 71 percent of participants were male, 29 percent were under 30 years old, 39 percent between 30 to 40 years old, 30 percent between 40 to 50 years old, and two percent between 50 to 60 years old. The majority of participants hold managerial organizational position (eight percent of non-supervisory staff, six percent junior manager, 63 percent middle manager, and 23 percent top manager) with average work experience of 13.2 years. Participants hold positions in both private (36%) and government (64%) organizations with average of 2,667 full-time employees.

Participants were asked to recall a recent project proposal they had proposed or reviewed and mark their level of agreement of the 15 items derived from Study 2 for that proposal. Proposed items (presented in random order) were phrased in form of statements on seven-point Likert scales from “Strongly disagree” (1) to “Strongly agree” (7).

Data Analysis

We purified the proposed measure by conducting an Exploratory Factor Analysis (EFA) to determine the dimension structure and items of the QTB construct. In order to examine whether the items load onto the specified dimensions, we performed a rotated Varimax factor analysis with Kaiser Normalization.

Results

The results of Exploratory Factor Analysis are presented in Table 4. Results confirm the belongings of all items in the three proposed dimensions. These three dimensions explained 63% of the variance. 12 of the 15 items fell into the exact same dimensions confirmed at the end of Study 2. The three other items were rephrased to improve their readability.

* Loading less than .40 are not presented.
  Note: Items #11 and #13 were removed in Study 2.

Table 4: Exploratory Factor Analysis.

These results support the proposed structure of QTB. The final QTB scale is presented in Table 5.

Table 5: The Quality of Target Benefits Scale.

Discussion

Project selection significantly affects organizational performance by ensuring effective allocation of organizational resources. Target benefits of a proposed project are an essential consideration in the project selection process (Young 2006). This paper presented results of three studies aimed at defining the new QTB construct and developing a scale for its assessment. In Study 1, interviews with senior managers in the Australian Public Sector confirmed dimensions derived from the goal setting theory literature, as well as contributed additional ones. In Study 2, content validity conducted with international experts confirmed the structure of the proposed structure. In Study 3, a quantitative survey was conducted to validate these dimensions and items. Results of Exploratory and Confirmatory factor analyses revealed that QTB can be operationalized as a three-dimensional 15-item scale. These dimensions are: specificity, attainability, and comprehensiveness.

These findings suggest that target benefits of a proposed project should fit into the organization’s strategic goals, be explicitly defined with clear and relevant measures and a target value, be achievable given the organization’s context and constraints, and be considered from a variety of aspects. As an illustrative example, we can refer to the comprehensive procurement process improvement case study discussed by Zwikael and Smyrk (2011). In this project, a well-formulated target benefit may be “Aligned with the strategy to enhance operational efficiency, National Procurement Manager is accountable for reduced procurement costs by 25% per order by one year from project approval, measured by the company’s cost accounting information system.” This target benefit should also take into account organizational capabilities and constraints following consultation with diverse groups of stakeholders, and be part of a web of target benefits (for example, includes both financial and non-financial benefits).

The proposed QTB construct has significant practical implications. At the most fundamental level, the dimensions and items of QTB can serve as a checklist to guide managers in formulating target benefits for new proposed projects. In addition, they can also provide senior managers with a more structured basis for making project selection decisions. Executives will also be more confident that funded projects are well aligned with organizational strategic goals (Gimbert, Bisbe, and Mendoza, 2010). This can resolve common problems of unclear project objectives, such as users’ needs which are misunderstood, or ignored, project objectives which are unknown or misunderstood, conflicts over objectives and/or strategies concerning the project, and lack of commitment to the project from key stakeholders (Klakegg, 2009). In addition, senior managers can develop an understanding on what target benefits are expected at project completion, how these benefits should be measured and who is accountable for their realization.

To further strengthen the value derived from the scale, organizations can develop norms, as recommended by Churchill (1979). In particular, organizations may set cut-off scores for the complete QTB scale or for each dimension, and introduce a gate-keeping system where project proposals are not discussed before a minimum quality threshold has been reached. These threshold levels can be decided following a benchmarking exercise. For example, QTB scores of each proposed project can be compared with other proposals within the organization (internal benchmarking), or a top-achieving organization (external benchmarking) (Meredith & Shafer, 2007). As a potential baseline for comparison, using the findings from our Study 3, the average scores for the complete QTB scale was 5.8 on a seven point scale. This suggests that if a proposed project’s target benefits receive an average score of five, they may still not satisfy the requirement and hence need to be further revised. In addition, organizations can set threshold levels for each QTB dimension as well as set weights to determine their relative importance. By developing organization-wide norms for QTB scale, the organization can ensure a certain level of uniformity in the assessment of QTB of new proposed projects, leading to a more informed and justifiable project selection decisions. The new process can also assist with increased levels of transparency in decision making (Mihm, 2010).

Conclusions

Project benefit management is an important area within the project management research. This paper focuses on target benefits formulation—the first and critical step for successful benefit realization (e.g., Bradley, 2010). This paper contributes to the literature by developing a reliable, valid, and replicable scale of QTB, to be used for further theory development. For example, in the area of performance evaluation, the construct may be used to empirically examine the impact of QTB on project performance, and in strategy to narrow the potential gaps between the organizational strategies and operations. For example, current strategy management theories excel in developing strategic plans and comparing proposed projects within the organizational portfolio to support agreed strategies. However, they fail to link organizational strategic objectives to target benefits, which represent individual project goals.

Our studies are not without limitations. First, participants in the three studies were recruited from five countries only. An expansion of our studies to organizations in a range of contexts and countries can enrich the validity and generalizability of the construct. Second, our research assumes all QTB dimensions are of equal weight. Future research can look into the impact of different weighting schemes for each of the scale dimensions. It is also important to acknowledge the fact that individuals at times deliberately inflate target benefits to increase the chance of project approval (Boardman, Mallery, & Vining 1994; Flyvbjerg, 2007; Jenner, 2009) and accept that this paper does not aim to solve this particular issue. Finally, it is widely recognized that projects vary (Shenhar, 2001). Our sample size did not allow us to examine the influence of the differences in project types and industries on the QTB scale.