Optimal schedule performance
what factors have the most impact on schedule performance?
Speedy delivery is almost always one goal of the project and often it is the primary goal or even a project constraint. Organizations also usually have a shorter time to market as one of their primary goals.
In order to work on improving duration, we need to answer the question: “What factors are most closely related to project duration?” After all, if we want to improve, we need to know where to focus our improvement efforts.
To answer this question, I mined the QSM database, specifically for projects that completed in the 21st century and had metrics that passed data quality checks. The techniques used are described in the Methodology section, including the computation of a standardized residual of duration. This standardized residual is used in this article to represent project duration.
Examination of a large number of candidate factors (40 quantitative variables) revealed two general interesting areas that correlated with the standardized residual of duration versus size:
- Overlap in months shows a correlation with duration; positive correlation means more overlap yields longer durations
- The various error metrics and reliability metrics show correlation with duration
- Error counts are positively correlate; more errors mean longer duration
- Reliability (mttd) is negatively correlated; higher reliability means shorter durations
- Error per unit of size is positively correlated; higher defect density yields longer duration
Phase overlaps occur, generally, because project and program managers are attempting to shorten the overall project duration by having concurrent phases. How interesting, then, that major phase overlaps are associated with longer durations!
The following two graphs have the months of phase 2 overlap (Functional Design) on the horizontal axis. This is the number of months that Functional Design overlapped with Main Build. The vertical scale is the standardized residual of duration, where higher values represent projects that had longer duration. The first graph is for Business applications and the second is for Engineering. The sloping lines represent a linear regression and a 95% confidence interval on the mean.
In general, longer overlaps (in calendar months) result in higher duration that would be predicted.
What does a typical project with high overlap look like, and how does it compare to a typical project with low overlap? The following table provides means and medians for the key metrics of effort, staff, duration, and size in addition to functional design overlap months. The projects have been divided into two groups, based on whether the functional design overlap months was higher or lower than the median.
|Super-group||Functional Design Overlap||FUNC Overlap (Months)||MB Effort (MM)||MB Duration (Months)||MB Peak Staff (People)||Effective SLOC||Standardized Residual (LogMBDur vs LogESLOC)|
To compare typical projects, the following graphs from SLIM Estimate use the median values for Business projects and for simplicity include only phases 2 and 3.
Compare the amount of overlap and compare the PI in these two typical projects. The first graphs are for a low overlap median project and the second graphs are for a high overlap median project. The PI for the low overlap is 14.5 and for the high overlap it is 11.1.
The second interesting correlation was quality factors. More errors result in longer durations, and correspondingly, higher reliability results in shorter durations. The following graph is the reliability (Mean Time to Defect) for business applications; again, there is a trend line, with a 95% confidence interval on the mean placed on the projects.
One interesting item to point out is that the trend line is under zero across the entire range of reliability. This is because the standardized residual was determined for all business applications, but those projects that reported MTTD had, on average, shorter durations than those that did not report MTTD.
“As the Japanese learned in 1950, productivity moves upward as the quality of process improves.” W. E. Deming
So, for Business applications, we see that the initial quality of the product is the key. In other words, higher quality results in shorter durations. If two products are of similar initial quality, we would expect one with higher quality to have a longer duration because it would have undergone more thorough testing and debugging.
This is not exactly the same for engineering applications. The correlation of MTTD and standardized residual is not significant for engineering. In the following two box plots, each box represents a quartile of MTTD, so that the box on the left is the 25% of the projects with the worst reliability, and the box on the right is the 25% of the projects with the best reliability.
Engineering is interesting, in a non-intuitive sort of way. Quality in engineering appears to be created by extending the duration (i.e., testing quality in), whereas in business the duration is more a direct result of the initial quality. In other words, in engineering systems, the quality requirement drives the duration by affecting the testing and debugging time.
Qualitative Assessment Factors
The assessment factors are on a scale of zero to ten, where zero means none, and ten means a high amount. Following an evaluation of all the assessment factors in SLIM, the assessment factors that exhibit the strongest correlations with duration prediction are described in this section.
Of course, assessment factors are qualitative, and assignment for a project is somewhat objective; however, it is still possible to derive general conclusions by grouping the ratings and looking at large differences.
Technical and communication complexity is important to the duration of business application development projects.
- Overall complexity is the overall technical complexity; higher numbers represent higher complexity. The coefficient is positive, meaning that higher complexity is longer duration.
- Team communication complexity is the level of team communication complexity. Higher numbers mean more complexity. The coefficient is positive, meaning that higher complexity is longer duration.
In the following box plot, overall complexity of 1 to 4 is “Low,” 5 to 8 is “Medium,” and 9 to 10 is “High.” The lowest complexity projects tend to have the shortest durations. The median business project with high complexity is 0.14 standard deviations above the duration trendline, whereas the median low complexity project is 0.83 standard deviations below the duration trendline. This is a difference of almost a full standard deviation.
Projects with low team communication complexity tend to have the shortest durations. In the following box plot, Team Communication Complexity of 1 to 4 is “Low” and 5 to 10 is “High.” Team communication complexity is a significant factor, although it does not have as strong an influence as overall complexity.
For engineering applications, three factors that had the highest significance to duration are:
- Design tooling is the capability of the design tool; 10 is high capability. The correlation is negative, so that higher capability results in shorter duration.
- Closeness arch limit is how close to the architectural limits is the development environment (memory, storage, etc.) The correlation is positive so that a higher closeness results in a longer duration.
- Construction tooling is the capability of the construction tool; 10 is high capability. The correlation is positive so a higher capability results in shorter duration.
Engineering projects with the best design tools tend to have shorter durations. In the following box plot, Design Tooling of 1 to 3 is “Low,” 4 to 6 is “Medium,” and 7 to 10 is “High.”
Durations became gradually shorter as Construction Tooling ratings increase from five to ten. Improving the tools from 1 to 6 makes little difference. In the following box plot, Construction Tooling of 1 to 6 is “Low,” 7 to 9 is “High,” and 10 is “Very High.” A typical engineering project with construction tools rated as 10 has a duration that is a full standard deviation shorter than the typical engineering project.
For engineering projects, as the system approaches the architectural limits, the duration increases. In the following box plot, Closeness to Architectural Limits of 1 to 2 is “Low,” 3 to 5 is “Medium,” and 6 to 10 is “High.”
A number of factors are frequently considered to be important in determining duration of software projects; among them are team skill levels and team size.
Although there is a significant relationship between average team size and duration, the log of the average team size explains less than 3% of the variation in the duration residual. Other articles in this Almanac look at the impact of staff size on various output measures.
Having a skilled and experienced team is certainly important for a number of reasons; however, team skill alone does not significantly impact project duration. The following table lists correlation coefficients (using a technique applicable for ordinal variables, such as the qualitative assessment factors). The significance for Overall Personnel and Staff capability is high enough to cast doubt into whether a relationship actually exists (generally, a significance of less than .05 is considered to be evidence for the existence of a relationship).
- Overall personnel is the overall capability of the personnel involved in the project, where 1 is low and 10 is high
- Staff capability is the capability and experience of the development team, where 1 is low and 10 is high
However, there do appear to be some staff factors that impact duration; for example, team motivation and management effectiveness has a relationship with duration. Although the relationships are weak, the significance factor is sufficient to provide evidence that the relationships are real.
- Team motivation is the level of motivation of the development team, where 1 is low and 10 is high.
- Management effectiveness is the effectiveness of management and leadership, where 1 is low and 10 is high.
Both correlation coefficients are negative, which means that, in general, as team motivation or management effectiveness increases, duration decreases.
|Correlation with Standardized Residual (LogMBDur vs LogESLOC)|
|Overall Personnel||Correlation |
|Staff Capability||Correlation |
|Team Motivation||Correlation |
|Staff Capability||Correlation |
In the following box plots, team motivation of 1 to 3 is low, 4 to 8 is medium, and 9 to 10 is high. Management effectiveness of 1 to 4 is low, 5 to 8 is medium, and 9 to 10 is high.
Project duration (i.e., time to market) is often an important constraint. Organizations that want to shorten their project durations should improve their processes. This article has highlighted some of the factors that have a major impact on project durations. Organizations should identify and focus on their processes that control these factors.
In order to shorten project durations, it is important to:
- Improve the upstream quality of the product (inject fewer defects into the constructed product)
- Improve testing efficiency (especially in engineering applications)
- Track and use measures of product quality
- Minimize the overlap between major phases (the four SLIM phases)
- Reduce technical complexity and communication complexity where possible (especially for business application projects)
- Improve tools for design and construction (especially for engineering application projects)
- Either improve the architecture or modify designs so that the engineering projects are not close to the limits of the architecture (memory, storage, speed, etc.)
- Keep the development team motivated and retain effective managers and leaders
The first step in mining the data was to create an output variable that measures project duration, which is required to search for candidate causal factors.
To begin, we divided the projects into supergroups (Business, Engineering, and Real Time), because we know that these three supergroups have different project duration distributions. In doing so, the Real Time group ends up with a sample size that is too small for this purpose; hence, this article focuses on Business and Engineering projects.
Next, we need to normalize the projects for size. We already know that larger projects tend to take longer, and we know that the relationship between size and duration is exponential. This size normalization will provide the output variable that will be used to identify impacting factors. To normalize the durations, we will use a regression of duration versus size.
To normalize by size, we computed the standardized residuals for the regression of log ESLOC versus log Main Build Duration. This numerical result is the number of standard deviations the project is above or below the regression line. On the following scatterplot of Business applications, duration is on the vertical axis (the dependent variable) and size is on the horizontal (the independent variable).
In this new residual variable, negative numbers represent the number of standard deviations that the project is below the predicted value; therefore, negative numbers are good and denote durations shorter than the regression predicts, whereas positive numbers denote longer durations. A value of +1 represents a project that had a duration that was one standard deviation above the trend line, a value of -1 represents a project that had a duration that was one standard deviation less than the trend line.
Since the residuals are standardized, they fit a normal distribution (as can be seen in the two histograms) and can now be used to compare durations of projects, regardless of project size. In the following histograms, a normal curve has been placed over the data bars.