Improved owner-contractor work relationships based on capital project competencies

Stuart D. Anderson, Associate Professor, Texas A&M University
Shekhar S. Patil, Doctoral Candidate, Texas A&M University
G. Edward Gibson Jr., Associate Professor, The University of Texas at Austin

Introduction

Organizations respond to the changing business environment by adapting to the demands of that environment. Particularly, in the case of owner organizations, any attempts to reduce costs and maintain profitability while delivering quality products and services are frequently accompanied by downsizing or reducing capital projects organizations, shifting capital project responsibilities to business units or operating facilities, or outsourcing more project work to contractors. Concurrent with this phenomenon is the gradual attrition, through retirement, of a whole generation of experienced managers and engineers. The combined effect of these changes has left many owners inadequately equipped to develop and execute capital projects effectively (Gibson et al., 1998; Sullivan et al., 1997). Therefore, owners are seeking ways to realign their relationships with contractors, while leveraging contractor manpower and expertise where necessary. A recent research effort attempted to provide a solution to this problem by developing a decision process to assist owner companies in creating and sustaining properly aligned owner-contractor work relationships. The decision process was termed as “Owner-Contractor Work Structure (OCWS) Process,” and was published by the Construction Industry Institute (CII) in 1997 (CII, 1997).

OCWS Process Overview

The OCWS process involves identification of key capital project competencies, determination of competencies that are core and non-core to the owner, and evaluates the sourcing approach for each competency. The sourcing approach is described in terms of a qualitative definition of the work relationship between an owner and a contractor, depending on the extent of their individual involvement in performing a competency for a project, or a capital program.

The term “owner-contractor work structure” that was coined in the CII research implies a set of competencies and corresponding work relationships. The term “contractor” may represent a design or construction contractor, a consultant, a supplier or an entity that provides services to the owner organization. The process was developed from an owner's perspective. A project-focused and a corporate process were developed, depending on whether sourcing decisions are made for a specific project or for a capital program. The key terms of the OCWS process are as follows:

Owner/Contractor Work Structure: The strategic distribution of roles and responsibilities between the primary project participants based on key project competencies, measured on a continuum of level of involvement of the owner and the contractor.

Competency: A project work process that is comprised of functions and associated critical capabilities needed to develop and execute a capital project.

Functions: Activities and tasks that describe the work involved in performing a competency.

Critical capabilities: The knowledge, abilities, skills and experience that are necessary to perform competency functions.

Core competency: A competency that must be performed in-house by the owner, and is critical to the success of capital projects.

Non-core competency: A competency that could either be outsourced or performed in-house by the owner, depending on the project circumstances.

Work relationship: A relationship that defines the extent of involvement of the owner and the contractor, in performing, leading, and/or providing input with respect to a competency.

It is important to review the definitions of each of these terms as well as the concept of a work relationship in order to understand how the OCWS process works. The owner-contractor work structure is characterized in terms of three types of work relationships between the owner and the contractor, and two other possibilities wherein either the owner or the contractor are fully responsible for performing the work. The work relationship typology used to describe roles and responsibilities in the work structure is illustrated in Exhibit 1. The definition of each work relationship is as follows:

OP:Owner performs all work involved in the competency using their resources and work process.

OP/CI: Owner performs all work using owner's work process with contractor input. The majority of the work is performed using owner resources. Contractor provides input or acts as a consultant.

OL/CP: Owner leads the work with contractor performing all detailed work using owner's work process. Owner leads by setting guidelines, directing, reviewing and approving work. Contractor performs most of the competency work functions according to owner's work process.

CP/OI: Contractor performs all work using contractor's work process with input from the owner. The majority of work is performed using contractor resources.

CP: Contractor(s) perform all work involved in the competency using their resources and work process. The owner can still supply input and guidance by performing a project management oversight competency.

The OCWS process includes detailed flow charts illustrating the inputs, actions, decisions and outputs of the process, and a set of worksheets that are used to document decisions and the rationale behind the decisions. Depending on whether the process is used for a specific capital project or for a capital program level application, the steps in the process are somewhat different.

A limitation in the development of the CII OCWS process was that the process was not fully validated. Therefore, there was minimal information available on how the industry would implement the process, and the extent to which owner companies found the OCWS process useful when managing a capital program or a specific project. Concerns regarding the lack of validation of the OCWS process provided a premise for undertaking follow-up research.

Research Methodology

The Construction Industry Institute and the Alfred P. Sloan Foundation jointly funded the follow-up research with the objective of testing, validating and improving the Owner/Contractor Work Structure process. Due to subjectivity inherent in an assessment of owner-contractor work relationships with the help of the OCWS process, the nature of this research was qualitative. Also, given the limited existing knowledge base on the determination of owner-contractor work structures, investigations in this area were exploratory in nature, making it difficult to construct hypotheses based on quantifiable parameters. Therefore, the research methodology relied on an owner survey, case studies and the Delphi method (Arditi & Ferreira, 1996; Linstone & Turoff, 1975). Three research phases were followed.

Phase I—Owner Survey

Phase I of the research covered a survey of owner companies in North America, the formulation of a research methodology, and the development and testing of instruments and proposals for the case studies that followed in Phase II. As part of Phase I of the research, owner companies were contacted in the summer of 1998 to study the use of the owner-contractor work structure process developed by the CII. The results of this survey established the premise for subsequent phases of this research (Anderson et al., 2000). Out of the total of 62 owner companies contacted, 23 responded to the survey. Of the 23 companies, only three had used the OCWS process and 11 had reviewed the process but not used it. The remaining nine owner companies had not reviewed the OCWS process even though they had acquired a copy of the publication from CII.

Phase II—Case Studies

The case studies in Phase II involved a combination of interviews and process applications for gathering data. To initiate the case studies, company-specific proposals were developed with the objective of addressing each company's particular problem through the implementation of the OCWS process. Two owner companies in North America accepted the proposal to implement the OCWS process. OCWS process implementation with one company involved the development of a new proposed alliance with a large contractor. In this case, the owner was a large refining company. The second implementation effort involved restructuring of the owner company's capital projects organization based on a business model approved by top management. In this case, the owner was a large power company. A third company agreed to participate in an interview to evaluate a recent application of the process for developing an owner-contractor work structure for an overseas chemical plant project.

Exhibit 1. Owner-Contractor Relationship Continuum

Owner-Contractor Relationship Continuum

The conceptual framework of the OCWS process was explained to participants in the case studies with the help of an owner-contractor work relationship continuum, as shown in Exhibit 1. The case studies provided valuable insight into the OCWS approach for developing owner-contractor work structures, and identified several weak areas in the existing process (Anderson et al., 2000). As a result, the OCWS process was modified and improved upon. The modifications in the OCWS process provided the necessary elements to help owner companies accomplish the following four key objectives:

1. Forming optimal work relationships with designers, constructors, suppliers and consultants.

2.“Rightsizing” in-house capital project development and execution capabilities.

3. Addressing the problem of loss of expertise due to attrition of experienced personnel.

4. Forming potentially successful strategic alliances.

The steps and associated worksheets in the improved OCWS process are illustrated in Exhibit 2. The worksheets provide a mechanism to document decisions and create a common language that can be used throughout the life cycle of capital projects.

Phase III—Delphi Validation of OCWS Process

Validating the OCWS process posed a unique problem. Unlike most research problems that involve the study of organizational phenomenon based on some available data or surveys/interviews of personnel, validating the OCWS process involved having the research participants assess various aspects of the process. The participants’ assessment of the process was based on their experiences with forming effective owner-contractor work relationships. Since the reliability of this approach was a function of how well the participants understood the process, it was inappropriate to conduct the validation in a single administration. A Delphi approach that involves multiple iterations of data collection was considered most suitable for testing the validity of the OCWS process that resulted from Phase II. The three key characteristics of a Delphi study are: (i) Anonymous response; (ii) Iteration and controlled feedback; and (iii) Statistical group response (Dalkey, 1969).

Prior to beginning the Delphi validation the participants were first selected from a wide range of organizations, based on the extent of their experience in the area of capital project development and execution. Each participant was provided with an overview of the modified owner-contractor work structure process, including Exhibit 2 and a structured response protocol. Ordinal Likert scales were used for soliciting responses from the participants, by having each of them rate various aspects of the OCWS process on a 1 to 7, Strongly Disagree to Strongly Agree scale. The responses were based on the participants’ knowledge of capital project development and execution, and the relevance of the process described in the overview. Reliability and validity issues were addressed by computing appropriate indices.

Exhibit 2. Steps in the OCWS Process and Process Worksheets

Steps in the OCWS Process and Process Worksheets

Exhibit 3. Composition of Participants’ Experience Level

Composition of Participants’ Experience Level

Two types of reliability were critical in the study. The first, known as test-retest reliability, measured the consistency of results on repeated trials. In this procedure, each person's scores on the first administration of the test are related to his or her scores on a second administration, to provide a reliability coefficient (Tuckman, 1999). The test-retest reliability was measured on a pilot basis prior to conducting the Delphi process. The second type of reliability measure that was relevant to this study was internal or inter-item consistency. The internal consistency was measured after each Delphi round.

Internal consistency was one of the two separate indicators that were used to draw inferences regarding the outcome of each Delphi round, and determining the need for a subsequent round. It was measured by computing the Intraclass Correlation Coefficient (ICC), which is a function of the true score variance and the error variance based on an Analysis of Variance (Bravo & Potvin, 1991; Kastein et al., 1993; Kozlowski & Hattrup, 1992; McGraw & Wong, 1996; Nunnally & Bernstein, 1994; Shrout & Fleiss, 1979). The ICC measures the degree of consistency for measurements that are averages of the k independent measurements on randomly selected participants. In this research, the ICC was based on a two-way random effects analysis of variance, which is mathematically the same as Cronbach's alpha in psychometrics (Cohen & Swerdlik 1999;McGraw & Wong, 1996).

The other indicator that was used to draw inferences regarding the outcome of each Delphi round was an Agreement Index (Kozlowski & Hattrup, 1992; James, Demaree, & Wolf 1984; James, Demaree,& Wolf 1993). The Agreement Index is a function of the ratio of observed or sample variance and the random measurement variance, based on a uniform distribution. Since high inter-observer agreement is not sufficient to insure the quality of the data collected, evidence of the reliability of the data was also necessary (Mitchell, 1979). Therefore, both the ICC and the Agreement Index were used to evaluate the data. It is important to note that the two measures are not related mathematically or otherwise.

The Delphi method served as an approach for testing the validity of the OCWS process. The ICC used to estimate internal consistency also provided evidence of construct validity (Cohen & Swerdlik, 1999). The multi-round process of Delphi enabled the identification of problems related to judgments of the participants, as well as possible biases involved in their opinions (Arditi & Ferreira Martins, 1996). The Delphi rounds were also very effective in resolving differences among the participants and addressing various issues that surfaced through participants’ comments during each administration.

Limitations

The decision to involve participants from owner companies only, and exclude any contractor companies from the research was based on the following considerations:

1. The objectives of the OCWS process are primarily geared towards fulfilling the needs of the owner. Therefore, this process is owner driven.

2. Contractors have an incentive to influence the owner into transferring as many competencies to the contractor as possible.

Notwithstanding these reasons, the fact that contractors were not involved in the validation effort is an important limitation of this research. Another limitation was the absence of data that would establish a correlation between the use of the OCWS process and capital project outcomes. Collection of such data was not possible due to the limited use of the process at the time of conducting this research.

Exhibit 4. Participants’ Familiarity With the OCWS Process

Participants’ Familiarity With the OCWS Process

Validation of OCWS Process

It was important to involve experienced project managers and executives in the validation effort. Invitation letters were sent to member companies of the Construction Industry Institute (CII) as well as non-CII companies that attended the 1999 CII Annual Conference. This group of companies represented a cross-section of the North American construction industry, and provided a source for experienced project managers and executives. In addition to these individuals, participants in the case studies that were conducted prior to the Delphi study were also contacted. Out of the ninety individuals who were invited to participate, forty-seven participated in the study.

The design of the Delphi study was based on a series of discussions with experienced researchers, and past research that used similar approaches for developing consensus (Arditi & Ferreira Martins, 1996;Hartman & Baldwin, 1995; Kastein, et al., 1993; Martorella, 1991; Raskin, 1994; Zinn, 1998). The design involved having each of the participants rate various aspects of the OCWS process, over successive administrations or rounds of the study. The degree of agreement among the participants and the reliability of their ratings was computed at the end of each round of the study. The agreement and the reliability that was achieved at the end of the second round, and the fact that no new issues emerged at that point suggested that a third round would not add any further value to the results.

Design of the Delphi Protocol

The first and foremost question to be addressed before commencing validation was what the participants needed to validate. This question was answered by developing research propositions, termed as “Themes,” that described what the OCWS process was designed to accomplish. The themes were defined as follows:

• Theme I: The Owner Contractor work relationship framework, illustrated in Exhibit 1, will enable project management to create optimal owner-contractor work relationships, given that all other conditions remain unchanged.

• Theme II: Steps in the OCWS process and the associated worksheets, illustrated in Exhibit 2, will assist owners and contractors in the creation of optimal work relationships, given that all other conditions remain unchanged.

• Theme III:Owner-Contractor work relationships based on this process will contribute positively to capital project performance, given that all other conditions remain unchanged.

• Theme IV:The knowledge-base created by using this process, that is, the core/non-core classification of competencies and corresponding owner-contractor work relationships, will help owner companies to mitigate the effect of knowledge lost by way of retiring experts and the loss of experienced personnel to other companies, given that all other conditions remain unchanged.

• Theme V: This process will be useful for creating and sustaining a potentially successful strategic alliance between two firms, given that all other conditions remain unchanged.

A second question to be answered was how to conduct the validation. It was answered by constructing multiple statements that corroborated the proposition of each theme, and then having the participants rate each statement on a one to seven point Likert scale where one represented the option “Strongly Disagree” and seven represented “Strongly Agree.”

Test-Retest Reliability of the Delphi Protocol

The response protocol for Delphi validation was tested for consistency of responses over time, by conducting two administrations within a gap of three weeks, to measure the test-retest reliability of the participant response protocol. Four individuals, including two graduate students and two faculty members at Texas A&M University participated in this exercise. Three of these individuals had more than five years of experience in planning and development of capital projects. The fourth individual had no such experience. Each person's scores on the first administration of the test were related to his or her score on the second administration, to provide a reliability coefficient. The Spearman rank correlation coefficient was used to measure the test-retest reliability, and compare the measured values with estimated critical values at 5% significance level (Tuckman, 1999). The test-retest reliability measurement helped identify unreliable statements that had to be modified or deleted from the response protocol.

Delphi Validation of the OCWS Process

A key characteristic of the Delphi approach is the iterative feedback from participants in the study. At the time of commencing the Delphi study, all participants were informed that they would be required to provide input over a minimum of two and a maximum of three iterations. The upper limit of three iterations was set considering both time and effort required by the participants, and past research which suggested that two to three iterations were sufficient for achieving satisfactory results in a Delphi study (Martorella 1991; Zinn 1998). The Agreement Index and the Intraclass Correlation Coefficient explained earlier, were used to draw inferences regarding the outcome of each Delphi iteration and determining the need for a subsequent iteration.

Exhibit 5. Summary of Round 1 Results

Summary of Round 1 Results

Exhibit 6. Summary of Delphi Round 2 Results

Summary of Delphi Round 2 Results

Background of Participants

From the 52 professionals who had initially agreed to participate in the Delphi validation, 47 professionals representing 32 owner companies and two consulting companies from North America participated in the first round. The participants had an average of 21 years of experience in the development and execution of capital projects, with a standard deviation of eight years. The composition of the participants’ expertise at various levels in the owner organization is shown in Exhibit 3. The terms “Executive Management,”“Project Management,”“Plant Operations,” and “Other” were used, based on a universally recognized distinction between levels of involvement in capital project work.

The data on the participants’ level of experience provided an important basis to split the participants into two subgroups: those working at the level of executive management, and those working closer to the level at which work is accomplished. Since individuals working at the latter level should be expected to have a substantially different perspective on using contract services (Gibson et al., 1998), splitting the data accordingly was considered very important. Moreover, the OCWS process has a two-level focus, one involving a project application and the other involving a corporate or capital program application. Depending on their level of experience, the participants could respond differently to questions pertaining to the process. The information about participants’ level of experience was used to assess whether or not differences existed in the perceptions of these two groups regarding various aspects of the OCWS process.

Another aspect of participant characteristics that was particularly important in this research was their familiarity with the CII OCWS process. As shown in Exhibit 4, over half the participants indicated that they had some prior knowledge of the OCWS process, published by the CII in 1997, under the title “Owner-Contractor Work Structure Process Handbook” (CII, 1997). This information was used to assess whether or not there were differences in the perceptions of those that were familiar with or had used the OCWS process, and those who had no prior knowledge of the process. This background was considered very important since it served to assess if there was any bias involved in the participants’ responses, due to their prior knowledge of the OCWS process.

Exhibit 7. Statistical Comparison of Means

Statistical Comparison of Means

Exhibit 8. Statistical Comparison of Means

Statistical Comparison of Means

Analysis of Round 1 Data

A summary of data obtained from the first administration of Delphi is shown in Exhibit 5. Each theme included between three and eleven statements that were rated by each of the forty-seven participants. The Agreement Index and the Intraclass Correlation Coefficient (ICC) were used to measure participant agreement and reliability of the data, respectively. The results showed general agreement among the participants for all the themes. However, low ICC for Themes I and IV, and a negative ICC for Theme V indicated that the results lacked consistency between the participants.

The descriptive statistics suggested moderate to general agreement based on the Likert scale. The mean ratings for all themes were between 5 and 6. The variability in terms of the standard deviation of participants’ responses ranged from 0.84 to 1.12. The standard deviation for individual statements under each theme helped in identifying the statements that showed a relatively high degree of variance among the participants.

The Agreement Index ranged from 0.87 to 0.97, clearly suggesting that there was substantial agreement among the participants on the five themes. For the Themes I, IV and V, the ICC values were unsatisfactory given a level of significance of 0.05. Also, since this was the participants’ first opportunity to evaluate the OCWS process, it was possible that they had different perspectives with regard to the process and what it could accomplish. To help resolve the differences in the participants’ perspectives to some extent and to provide them with an opportunity to reconsider their first round ratings in light of the knowledge they could gain from the ratings and comments made by other participants in Round 1, another round of Delphi was used.

Exhibit 9. Results of Delphi Round—Analysis by Subgroup for Themes I, IV,& V

Results of Delphi Round—Analysis by Subgroup for Themes I, IV,& V

The second round was conducted using the same response protocol in Round 1, except the addition of one statement under Theme II. The addition of the statement was based on comments made by some participants, which indicated that the strategic basis for core/non-core competency decisions had to be emphasized. In addition to the response protocol, each participant was provided with an assessment of the findings of Round 1, and a comparison of his or her responses with the group response.

Analysis of Round 2 Data

Some attrition of the participant pool is expected in a Delphi study, due to the limited time frame in which the participants are expected to read and understand Round 1 results, and complete the Round 2 response protocol. In this study, the number of participants in Round 2 dropped to forty-two.

Round 2 of the Delphi validation showed an improvement in the rating of various elements of the process by the Delphi panel. The results of Round 2 are summarized in Exhibit 6. These results suggested that the participants in the Delphi study changed their earlier response in Round 1 to some extent based on the knowledge they gained from Round 1 results. The mean ratings did, however, indicate that on a one to seven scale, the mean was above 6.0 only in the case of Theme III. On all other themes the mean ratings were between five and six, suggesting that the agreement was conditional. This was not a cause of concern though, since each user organization is very likely to adapt the OCWS process to fit within its own business approach.

The mean ratings for all themes increased marginally while the standard deviations reduced in Round 2. These data suggest that the panelists rated the process more favorably in the second round, while moving closer to each other on the rating scale. The ICCs improved in Round 2 over the values in Round 1, with the exception of Theme IV. This theme dealt with the value and the utility of the information created by using the OCWS process for future projects. On this theme, some participants expressed their skepticism about the value of such information for future projects since, in their opinion, it constitutes an initial plan only. Moreover, this knowledge may not be useful since staffing and training decisions are driven by factors that may change over time. This realization may have persuaded the participants to change their response in the second round, leading to an increased inconsistency in responses to Theme IV.

The reliability of the results of Round 2, measured in terms of the ICC, improved in comparison with that of Round 1 for Themes I and V, stayed the same for Theme III, reduced marginally for Theme II and reduced substantially for Theme IV. Theme II focuses on the steps in the OCWS process and the associated tools. Theme III focuses on the contribution that using the OCWS process makes, to improving capital project performance. Although both agreement and reliability were satisfactory for Themes II and III, the reliability indicated by the ICC for Themes I, IV, and V was not satisfactory. Theme I focused on the owner-contractor work relationship framework, Theme IV considered the utility of knowledge base created by using the OCWS process, and Theme V dealt with the applicability of the OCWS process for creating alliances. The low reliability for Themes I, IV, and V indicated that the participants remained inconsistent in their responses to these themes despite having two opportunities to make their decision.

Additional analysis of Themes I, IV and V concentrated on participants’ familiarity with the OCWS process prior to this study, and the extent of their executive management experience. In order to assess if the participants’ familiarity with the process influenced their ratings, the participants were segregated by the criteria of familiarity with the OCWS process. A statistical comparison of the means for each of these two subgroups, presented in Exhibit 7, indicated that the means were statistically not different, suggesting that participants’ familiarity with the OCWS process did not influence their rating of the process. This was an important finding since participant familiarity with the process would have biased the results.

The possible influence of the participants’ level of management experience was evaluated by stratifying the participants into two subgroups. The first group included 22 participants with more than two years of executive management experience. The second group included 20 participants with no executive management experience. These individuals were primarily involved in engineering, project management, and plant operations. Individuals in both groups did, however, have an average of 10 years of project management experience. Exhibit 8 provides a statistical comparison of mean ratings of the two subgroups.

Since the t-statistic for Themes IV and V was higher than the t-statistic at a significance level of 0.05 for a two-tailed analysis, the means were significantly different between the two subgroups, for these themes. This result suggests that the level of participants’ experience had some influence on their ratings for these themes. A likely explanation for this difference was that individuals without executive management experience were relatively more concerned about the influence of project circumstances and conditions, on decisions about capital projects. Since differences in mean ratings for the two subgroups were found to be statistically significant as shown in Exhibit 8, further analysis was deemed necessary to test the reliability of responses within each subgroup for Themes IV and V. The results of this analysis are shown in Exhibit 9.

The ICCs indicated that the reliability was reasonably high for the first subgroup in Theme I, and for both subgroups in Theme V. The p-values were also satisfactory for the first subgroup under Theme I and the second subgroup under Theme V, suggesting that the probability (on repeated trials) of obtaining ICC values at least as high as 0.62 and 0.74, respectively, was higher than the significance level of 95 percent. This probability was 93 percent for the first subgroup under Theme V, which was marginally lower than the significance level of 95 percent that is typically used in scientific research. The reliability coefficient for the first subgroup in Theme IV was fairly low, while the reliability coefficients for the second subgroup in Theme I and the second subgroup in Theme IV were very low.

Based on these computations, the data suggested that the executive managers were consistent in their response to Theme I, while those without any executive management experience were not consistent. One possible explanation for this finding related to broader perspective of executive managers, placing them in a better position to understand the conceptual framework of OCWS. In case of Theme IV, the two subgroups as well as the whole group are not consistent in their responses. This finding is indicative of an inherent flaw in the proposition of Theme IV, that the knowledge-base created by using the OCWS process will help owner companies mitigate the effect of knowledge lost by way of retiring experts and the loss of experienced personnel to other companies.

Each of the two subgroups provided consistent responses to Theme V. However, the responses were not at all consistent when the entire participant pool was evaluated together. One possible explanation for this inconsistency could be that the two groups viewed alliances differently, given that the term “alliance” was not defined in the response protocols. The participants’ comments did not offer other explanations for the inconsistent overall response to Theme V.

In summary, the data provided evidence to suggest that based on reliability considerations, Theme IV could not be considered valid. Also, Themes I and IV evoked a different response from those with executive management experience and those without such experience.

Results of the Delphi Study

The Delphi rounds suggested that some propositions or themes about the applicability and usefulness of the process were valid, while others were not entirely valid. Although this observation is based on the assumption that the statements under each theme were accurate measures for the respective themes, no attempt was made to introduce new statements for two reasons: (i) Comments made by the participants in support of their numeric ratings did not suggest the need for introducing new statements; and (ii) The Agreement Indices were reasonably high for all themes, including the ones with low or negative ICCs. The following results provided the evidence regarding validity of the five themes:

• The mean ratings for each theme ranged from 5.46 to 6.07, suggesting that the participants were in moderate to complete agreement with the themes.

• Agreement indices for the five themes were between 0.96 and 0.98, with the exception of Theme V that had an agreement index of 0.88, which is also very high.

• The Intraclass Correlation Coefficient was 0.86 and 0.96 for Themes II and III respectively. Although the ICC was low and unacceptable for Themes I, IV and V, additional analyses identified satisfactory ICCs for one subgroup under Theme I and both subgroups under Theme V. Theme IV, however, failed the reliability criteria.

• No new issues that would lead to changes in the OCWS process steps or tools emerged in the Delphi rounds. This suggested that the participants did not envisage further changes to the OCWS process.

These results eliminated the need for conducting another Delphi round or for making any changes in the OCWS process steps and tools. Therefore, it was concluded that with the exception of Theme IV, the OCWS process constituted a valid approach for creating and sustaining well-aligned owner-contractor work relationships.

Conclusions

The Delphi approach for validating the OCWS process led to the following conclusions:

• Theme II was validated. The steps and tools in the OCWS process can assist owner companies in the formation of optimal work relationships with designers, contractors, suppliers and consultants.

• Theme III was validated. Using the OCWS process is likely to contribute positively to capital project performance.

• Non-executive managers were not consistent in their response to Theme I. This suggests that it may be more difficult for this group to relate to the OCWS conceptual framework, since they may not have a top management perspective.

• Theme IV could not be validated. Therefore, the OCWS process may not help owner companies mitigate the effect of knowledge lost due to retiring experts and the loss of experienced personnel to other companies.

• Theme V evoked consistent responses from executive managers and others. However, the two groups were not consistent when combined together. Therefore, executive managers and non-executive managers differ in their perspective on the use of OCWS for structuring alliances. Their comments did not identify the specific issues that might have caused the ICCs to be different for the two subgroups.

In summary, the OCWS process constitutes a valid approach to encourage discussion, consensus, definition and decisions while developing owner-contractor work relationships for capital projects. It may not, however, help owner companies mitigate the effect of knowledge lost by way of retiring experts and the loss of experienced personnel to other companies. In terms of its suitability as a process for creating and sustaining potentially successful alliances, executive managers and project managers differed in their perspectives suggesting that the proposition was conditional and hence, could not be validated conclusively. Given the problems with understanding the implications of an outsourcing relationship, it is evident that an approach such as the OCWS process would improve the development and execution of capital projects.

Anderson, Stuart D., Patil, Shekhar S., Gibson, & G. Edward, Jr. (2000). Developing optimal owner-contractor work structures—case studies. Center for Construction Industry Studies, The University of Texas at Austin.

Arditi, D.,& Ferreira Martins, E.A. (1996). Acquisition, representation and validation of subjective knowledge.Microcomputers in Civil Engineering 11 (4), 239–248.

Bravo, Gina, & Potvin, Louise. (1991). Estimating the reliability of continuous measures with Cronbach's alpha or the intraclass correlation coefficient: Toward the integration of two traditions. Journal of Clinical Epidemiology 44 (4/5), 381–390.

Cohen, Ronald J., & Swerdlik, Mark E. (1999). Psychological testing and assessment.Mountain View, CA:Mayfield Publishing Company.

Construction Industry Institute. (1997). Owner/Contractor work structure process handbook. (Implementation Resource 111–2). University of Texas at Austin.

Dalkey, Norman C. (1969). The Delphi method: An experimental study of group opinion (RM-5888-PR). The Rand Corporation.

Dalkey, N.C., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management Science 9 (3), 458–67.

Gibson, G. Edward Jr., Davis-Blake, Alison, Broschak, Joseph P., & Rodriguez, Fernando J. (1998). Owner/contractor organizational changes— Phase I report Sloan Program for the Construction Industry, The University of Texas at Austin.

Hartman, Francis T.,& Baldwin, Andrew. (1995). Using technology to improve Delphi method. Journal of Computing in Civil Engineering 9 (4), 244–249.

James, Lawrence R., Demaree, Robert G.,& Wolf, Gerrti. (1993). rwg: An assessment of within-group interrater agreement. Journal of Applied Psychology 78 (2), 306–309.

James, Lawrence R., Demaree, Robert G., & Wolf, Gerrti. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology 69 (1), 85–98.

Kastein, Marcel R., Jacobs, Max, Van Der Hell, Roelof H., Luttik, Kees, & Touw-Otten, Fransja W.M.M. (1993). Delphi, the issue of reliability. A qualitative Delphi study in primary health care in the Netherlands.Technological Forecasting and Social Change 44, 315–323.

Kozlowski, Steve W.J., & Hattrup, Keith. (1992). A disagreement about within-group agreement: disentangling issues of consistency versus consensus. Journal of Applied Psychology 77 (2), 161–167.

Linstone, Harold A., & Turoff, Murray, (Eds.). (1975). The Delphi method: Techniques and applications. Addison Wesley Publishing Company.

McGraw, Kenneth O., & Wong, S.P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods 1 (1), 30–46.

Martorella, Peter H. (1991, Winter). Consensus building among social educators: A Delphi study. Theory and Research in Social Education, XIX, 83–94.

Mitchell, Sandra K. (1979). Interobserver agreement, reliability, and generalizability of data collected in observational studies. Psychological Bulletin 86 (2), 376–390.

Nunnally, Jum C., & Bernstein, Ira H. (1994). Psychometric theory. New York:McGraw-Hill, Inc.

Raskin, Miriam S. (1994, Winter). The Delphi study in field instruction revisited: Expert consensus on issues and research priorities. Journal of Social Work Education 30, 75–89.

Shrout, Patrick E., & Fleiss, Joseph L. (1979). Intraclass correlations: Uses in assessing interrater reliability. Psychological Bulletin 86 (2), 420–428.

Sullivan, George R., Yupari, Miguel A., & Anderson, Stuart D. (1997). Owner/contractor work structure: A process approach, (Research Report 111-11). The University of Texas at Austin, The Construction Industry Institute.

Tuckman, Bruce W. (1999).Conducting educational research, 5th Ed. Fort Worth: Harcourt Brace College Publishers.

Zinn, Jacqueline. (1998). Use of the Delphi panel method to develop consensus on laboratory performance indicators. Clinical Laboratory Management Review 12 (2), 97–105.

This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI or any listed author.

Proceedings of PMI Research Conference 2000

Advertisement

Advertisement

Related Content

Advertisement