A summary measurement of engineering productivity at the project level
University of Texas at Austin, Department of Civil, Architectural, and Environmental Engineering
Stephen R. Thomas,
Construction Industry Institute
Since 2002, the Construction Industry Institute (CII) has been working to develop a standardized engineering productivity metric system (EPMS) for benchmarking purposes. In this system, engineering productivity is defined as a ratio of direct engineering work hours to the engineering outputs, as measured by issued for construction (IFC) quantities. The EPMS consists of six major engineering disciplines with a number of underlying metrics. Engineering productivity can be accordingly benchmarked at any of these levels; however, there is a lack of project-level engineering productivity. The challenge is that IFC quantities are measured with different units and thus are difficult to advance to the project level. To overcome this barrier, this study examines three approaches for aggregating engineering productivity metrics to the project level, based on 112 heavy industrial projects. The selected project level engineering productivity measurement best summarizes the underlying engineering productivity metrics and provides a macro view of engineering performance; it allows owners and engineering organizations to benchmark engineering productivity at the project level and also lays the foundation for future engineering productivity analysis and research.
Productivity reflects how efficiently the major resources are used to produce the outputs. Improving productivity usually leads to competitive advantage and more profit; therefore, extensive research has been conducted to improve productivity in the construction industry. However, most productivity research has focused on construction productivity, and engineering productivity has rarely been studied.
CII undertook the development of the engineering productivity metrics system (EPMS) to support productivity benchmarking and research. The EPMS consists of engineering productivity metrics defined as a ratio of direct work hours to issued for construction (IFC) quantities, and productivity data are collected at the discipline level and below. However, there is a lack of a project level engineering productivity measurement (PLEPM) to summarize an overall status of engineering productivity of the discipline or lower level, which is primarily because IFC quantities of the different disciplines are measured by different units. For example, concrete quantity is measured by cubic yard, and structural steel is measured by ton. Accordingly, this paper presents the efforts to develop a PLEPM for benchmarking, as requested consistently by the industry.
Engineering productivity is defined as a ratio of input to output. The inputs of engineering may be clearly defined, but measuring the outputs is elusive (Sacks, & Barak 2008).
Various engineering productivity measurements have been used in previous research. For example, Thomas (1999) measured engineering productivity using hours per drawing, and Song, Allouche and Abourizk, (2003) used hours per designed element (a column or beam) to measure engineering productivity. CII (2001) utilized hours per engineered quantity (e.g., ton of steel or linear foot of pipe) to measure engineering productivity and explored influence factors such as engineering input quality and complexity. The analyses were only based on piping engineering because of data availability. Nevertheless, CII (2001) enumerated several advantages of quantity-based engineering productivity measures: (1) it is directly tied to the engineering activity; (2) it is less subject to manipulation; (3) both the inputs and outputs are already tracked in most current engineering environments; (4) it measures engineering and construction productivity on the same basis; and (5) it focuses attention on the final product rather than the intermediate product.
Based on extensive literature review and industry inputs, the CII benchmarking program defines engineering productivity metrics as a ratio of direct engineering work hours to IFC quantities in the EPMS (Kim, 2007). To reduce the data collection effort and to provide flexibility for engineering productivity benchmarking, the EPMS is organized into a hierarchical structure, as shown in Exhibit 1. Engineering productivity data (both work hours and IFC quantities) can be collected and compared at the major category, sub-category, or element level. In this hierarchy, engineering productivity metrics can be advanced from the element or sub-category level up to the discipline level, such as total concrete, total steel, and total piping. The EPMS was validated through initial data collection and preliminary analysis (Kim, 2007). However, since the launch of the system, there has been an increasing demand from the industry for a PLEPM that provides managers with a macro-view of engineering productivity. Furthermore, the lack of PLEPM inhibited understanding of engineering productivity, because the effects of best practices, project performance, and engineering productivity cannot be quantified. Thus, the development of PLEPM is imperative.
Exhibit 1 – The EPMS Hierarchy (Kim, 2007)
The construction industry uses a number of indices to summarize project performance. For example, safety performance can be measured by the total recordable incident rate (TRIR) and days away, restricted, and transferred rate (DART). A number of approaches have been used to develop productivity summary indices.
The first approach is based on the earned-value concept, which compares actual performance with expected performance. For example, CII (2004) proposed an engineering productivity measurement model as a ratio of actual work hours to predicted work hours. However, this approach is likely unreliable due to the large standard error of the estimated hours (Kim, 2007). Ellis and Lee (2006) defined a project-level productivity (PLP) index using total work hours divided by equivalent work units (EWU) to monitor multi-discipline daily labor productivity. However, this approach normalized installed quantities without considering their variances. Another approach standardizes underlying metrics and then aggregates them. As to metric standardization, Maloney et al. (1995) transformed variables with different scales ranging from 0 to 1 in order to make comparisons across the variables. Furthermore, the z statistic (z-score) is another common method used to standardize variables with the consideration of both sample mean and standard deviation (Agresti, & Finlay, 1999). As to metric aggregation, the weighted-sum of underlying metrics is a common method addressed in previous studies. Ibbs (2005) defined the end-of-project productivity index as change-impacted and change-unimpacted periods weighted by their work hours.
The EPMS Database
CII benchmarking is a member service, and traditionally CII members are primarily associated with heavy industry; therefore, CII’s engineering productivity dataset is heavily industry oriented. In the engineering productivity dataset, 112 heavy industrial projects were collected from 2002 to 2007. Exhibit 2 summarizes the distribution of these projects by various characteristics, including respondent type, project type, project nature, and project size. By respondent type, contractors submitted 82% of the projects. By project type, process projects such as chemical manufacturing facilities or oil refining plants dominate the database; addition and modernization projects account for 80%, and large (>$5MM) projects are approximately 61% of the database.
Exhibit 2 – Projects with Engineering Productivity Data by Characteristics
Three approaches: the earned-value method, z-score method, and the max-min method are examined to construct the PLEPM. Then, the developed indices are evaluated according to how well the indices summarize the underlying metrics.
The Earned-Value Method
This method calculates an engineering productivity index as a ratio of total actual work hours to total earned (predicted) work hours for all underling metrics. Predicted hours are equal to norm values for engineering activities multiplied by the IFC quantities. A simplified example with only concrete and steel disciplines is presented in Exhibit 3 to demonstrate the process.
Exhibit 3 – Development Process Using the Earned-Value Method
An index value of less than one indicates better productivity, because the engineering work took fewer hours than the baseline. Accordingly, for the pth project, the project level engineering productivity index (EVp) can be calculated with the earned-value method, as shown in Equation 1:
WHpi = the direct work-hours of the ith underlying metric of the pth project,
= average engineering productivity of the ith underlying metric, and
Qtypi = the IFC quantity of the ith underlying metric in pth project
The Z-Score Method
To improve the normality of the metric distributions, this approach begins by transforming raw engineering productivity metrics with a natural logarithm function before converting them into their z-scores, because previous research has verified that productivity metrics are non-normally distributed (Zener, 1968). Shapiro-Wilk test results showed that CII engineering productivity metrics in raw scale are non-normal distributed (p < 0.05), whereas transformed metrics are normally distributed (p > 0.05). Next, the transformed engineering productivity metrics are standardized to z-scores. Lastly, all the standardized underlying engineering productivity metrics are weighted by the work hours to roll up to a PLEPM. The following example in Exhibit 4 illuminates the index development process using the z-score method.
The index will range from -3 to 3. A negative value of the Zp index indicates engineering productivity is better than average. Given the ith underlying engineering productivity metric of the pth project (EPpi), EP’pi represents its natural logarithm transformation. Next, a standardized z-score, Zpi, is shown in Equation 2:
Zpi = z-score of the ith engineering productivity metric in the pth project,
= mean of natural logarithm transformed engineering productivity of the ith underlying metric, and
σEPi’ = standard deviation of natural logarithm transformed engineering productivity of the ith underlying metric.
Exhibit 4 – Development Process using the Z-Score Method
Finally, standardized underlying metrics can be aggregated using their work hours as the weights to represent engineering productivity at the project level:
WH pi = the direct work-hours of the ith underlying metric of the pth project
The Max-Min Method
This method standardizes the engineering productivity metrics by using the max-min approach and then weights the underlying metrics by their work hours. Exhibit 5 demonstrates an example for index construction using the max-min method for concrete and steel engineering productivity.
Exhibit 5 – Development Process using the Max-Min Method
The index will range from 0% to 100%. The smaller the index value, the better the productivity because it is closer to the project with the best engineering productivity. Therefore, for a given pth project, EPpi represents engineering productivity of the ith underlying metric, and Npi represents its normalized score, as shown in Equation 4:
Next, a work-hour–weighted average of all underlying normalized metrics of the pth project, Mp, can be calculated, as shown in Equation 5:
WH pi = the direct work-hours of the ith underlying metric of the pth project
Finally, these three approaches should be evaluated according to how well the underlying metrics are presented. In order to achieve meaningful benchmarking results, the projects should be benchmarked with the most comparable ones with similar characteristics, such as similar project size or project nature. Therefore, when evaluating the three approaches, projects were also grouped by their characteristics: Project size (<$5 million vs. >$5 million), project type (process vs. non-process), project nature (grassroots, addition, and modernization), work involvement (engineering only vs. engineering and construction), contract type (lump sum vs. cost reimbursable), and project priority (schedule driven vs. non-schedule driven).
The first step is for each project to calculate the percentile for all underlying engineering productivity metrics and the PLEPM in each project characteristic subgroup. Then, under each subgroup, a project has a weighted average percentile, which is equal to the percentile of underlying metrics weighted by their work hours. The next step is to calculate absolute difference, PD, between the percentile of PLEPM and the weighted average percentile. PD calculation is demonstrated in Equation 6. The lower the PD, the better the index represents its underlying metrics.
PDP = the percentile difference between PLEPM and the weighted average of underlying metrics of the pth project
WHpi = the direct work hours of ith underlying metrics of the pth project
Ptpi = the percentile of ith underlying metrics of the pth project
PtpPLEPM = the percentile of the PLEPM of the pth project
A simplified example illustrates how PD of a grassroots project is derived using only two underlying metrics, concrete and steel. First, extract all grassroots projects from the entire database. Considering all grassroots projects, this project’s engineering productivity percentiles for concrete, steel, and the PLEPM are calculated as x%, y%, and z%, respectively. Second, PD is calculated as shown in Equation 7:
Finally, for each project characteristic, the can be calculated as the average of all projects with the examined characteristic. Results are shown in Exhibit 6. Regardless of project characteristics, of the z-score index is consistently better than the earned-value and max-min indices, with differences of 0.98% and 2.56% on average, respectively. These outcomes suggest that the z-score method is the best candidate to produce the PLEPM.
Exhibit 6 – Average PD of Project Index by Characteristics (*The best among the three approaches)
Conclusion and Path forward
Using EPMS to benchmark engineering productivity provides meaningful insight on how engineering is performed at different detail levels and helps project managers to recognize the potential improvement. On the other hand, it is also important to provide a macro-view of engineering productivity. However, due to different units of engineering productivity metrics, it’s difficult to aggregate engineering productivity to the project level. This study examined three approaches to construct a PLEPM for benchmarking purpose. The results suggest that the z-score method is superior to the other approaches because the index best represents their underlying metrics. However, it is also noted that the z-score method may have the deficiency in that it’s hard for industry practitioners to interpret the index value.
A reliable PLEPM using the z-score method opens new opportunities for engineering productivity benchmarking. The index can be used to analyze effects on engineering productivity from the use of best practices, such as front-end planning, constructability, or change management at the project level. Nevertheless, based on the inputs from the industry, it’s also critical to use the index to track the trend of engineering productivity.
Agresti, A., & Finlay, B. (1999). Statistical methods for the social sciences, Prentice Hall, Inc., Upper Saddle River, NJ
Construction Industry Institute (CII). (2001). Engineering productivity measurement. RR156-11, Construction Industry Institute, The University of Texas at Austin, Austin, TX.
Construction Industry Institute (CII). (2004). Engineering productivity measurements II. RR192-11, Construction Industry Institute, The University of Texas at Austin, Austin, TX.
Ellis Jr, R. D., & Lee, S.-H. (2006). Measuring project level productivity on transportation projects. Journal of Construction Engineering and Management, 132(3), 314–320.
Ibbs, W. (2005). Impact of change’s timing on labor productivity. Journal of Construction Engineering and Management, 131(11), 1219–1223.
Kim, I. (2007). Development and implementation of an engineering productivity measurement system (EPMS) for benchmarking, Ph.D. Dissertation, The University of Texas at Austin, Austin, TX.
Maloney, W. F., & McFillen, J. M. (1995). Job characteristics: Union-nonunion differences. Journal of construction engineering and management, 121(1), 43–54.
Sacks, R., & Barak, R. (2008). Impact of three-dimensional parametric modeling of buildings on productivity in structural engineering practice. Automation in Construction, 17(4), 439–449.
Song, L., Allouche, M., & Abourizk, S. (2003). Measuring and Estimating Steel Drafting Productivity. Honolulu, HI., 67–75.
Thomas, R. (1999). Conceptual model for measuring productivity of design and engineering. Journal of Construction Engineering and Management, 5(1), 1–7.
Zener, C. (1968). An analysis of scientific productivity. Proceedings of the National Academy of Sciences of the United States of America, 1078–1081.
©2010 Pin-Chao Liao, Stephen R. Thomas
Originally published as a part of Proceedings PMI Global Congress 2010 – Washington, DC