How Intel models project complexity … A model you can actually use
Teams at Intel are using a relative model to assess the complexity of new projects. This model is based on known drivers of project effort and duration, backed up by historical data. Using this model, teams are able to predict, with a high degree of accuracy, the total headcount and project duration. The analysis is quick to complete and flexible enough to support “what if” modeling so that the team can arrive at a set of project parameters that best fits the available resources and desired timeframe. This paper will discuss how the model was constructed, how historical project performance data is used to validate and refine it, limitations and opportunities of the model, and finally, how Intel teams are using it today.
The complexity model is a combination of known project complexity drivers, a relative scale, and a calculation of the project complexity score. This paper will discuss how the complexity drivers are identified using brainstorming techniques and subject matter experts, the logic and derivation of the relative scale, and the calculation of the project complexity score. This paper will include specific considerations, calculations, and scoring techniques for building the model.
Historical data forms the data set that makes the model useful. The model is validated and refined by scoring previous projects with known resource levels and duration. In this way it is possible to correlate complexity scores with an average headcount level and project duration. The results are then used to improve resource planning and pipeline management.
At Intel we have teams actively using this methodology to assess relative complexity across a portfolio of projects. Here we will share actual complexity models that have been developed and the data sets built with historical data to inform the model.
This model does have some limitations, however. Since it is based on historical performance, its usefulness for analyzing novel or radically different projects is constrained. Further, complexity drivers change over time as a team’s skills and experiences grow, thereby requiring periodic updates to the model. Along with the limitations comes interesting opportunities to expand the applicability of the model. Here we discuss ideas for adapting the model to an agile methodology and the expansion of the model to analyze programs in order to understand the aggregated resource requirements and expected program duration.
The complexity model enables fast, high confidence estimates of project resource requirements and duration. It can be developed in a short period of time, used to evaluate new opportunities quickly, and provides a mechanism for optimizing the overall project pipeline through “what if” scenario analysis. This paper will also discuss how real teams are using this model to drive predictability and improve accuracy in the scoping analysis of new opportunities.
As the world we live in becomes increasingly faster, smaller, and more interconnected, projects have become more complex and by extension more ambiguous. This is not news to project managers, as we’ve seen project complexity focused research and conference tracks devoted to this very problem. Where once it was a matter of discipline to understand the business drivers and make informed tradeoffs, today’s project manager must juggle a multitude of possible options for project execution. The tradeoffs which were once simple are now imbued with the morass of complicating factors like privacy concerns, digital security, regulatory compliance, and the inherent politics that come with large, multinational corporations. Further, today’s project teams are dispersed geographically, virtual, and made up of more generations than ever before. Take for instance, the project communication plan. Today team members communicate in a wide variety of mediums often with geographically induced time delays. The old communications methods don’t work so well and the newer communication technologies have not been widely adopted. The project manager now spends more time evaluating new communication technologies and drafting effective communication plans than ever before. As the number of options for communicating increases, the decisions as to which technologies to use becomes more complex and at the same time more ambiguous. Do I switch to a digital forum for disseminating project updates? What about the third party vendors we are working with, can they access that forum externally? The younger team members do most of their communication via instant messaging, so how do I loop them in, if I change the communication channel? What should be a straightforward decision about how to communicate project status has become more difficult to wade through.
Another area where this increasing complexity has had an impact is in the initial scoping of potential new projects or opportunities. At this early stage of analysis, the business drivers, design intent, and some high level concepts of execution are all that the team has to make an assessment about schedule, scope, and resources needed to deliver this new product or capability. Project scoping is something we know how to do and there’s a depth of published knowledge to draw upon. The challenge now is a bit different as the number of options for design and execution are larger and the business drivers are more fluid, resulting in a substantial number of permutations of options for delivering the project objectives. How do you comply with the privacy regulations in the U.S., the EU, and China simultaneously…in an environment of constant change? The applications we are developing need to run on Windows, Android, iOS, and who knows what other operating system will bubble up next and need to be supported? Users of public spaces want more connectivity and customers everywhere are expecting more immersive experiences so how do you determine which technology to implement today will have the longevity to remain relevant throughout the lifetime of the venue? Again, as the number of options expands, the environment continues to be driven by increasingly disconnected geo-political regulatory and compliance factors, and as the pace of innovation accelerates, scoping projects becomes much more complex for project teams around the globe.
At Intel, we too are scoping increasingly more complex projects, and this paper discusses one tool that has allowed teams to utilize a relative model to assess the complexity of new projects. This model is based on known drivers of project effort and duration, backed up by historical data. Utilizing this model, teams are able to predict, with a high degree of accuracy, the total headcount and project duration. The analysis is quick to complete and flexible enough to support “what if” modeling so that the team can arrive at a set of project parameters that best fit the available resources and desired timeframe.
Constructing the Model
The objective for creating this model was to be able to compare and contrast the effort required to complete unrelated projects. The underlying assumption of the model is that highly complex projects require more effort to execute than their less complex counterparts. Therefore a model that can assign a relative score based on the expected complexity of a project can be used to compare dissimilar projects. Paired with historical data, this model can be used to extrapolate an expected effort to execute the project and predict the overall project duration.
Identifying the Complexity Drivers
To begin constructing the model we first worked to identify what project characteristics correlated to significant effort. These characteristics are the complexity drivers for project execution. A group of subject matter experts (SMEs) from the functional areas impacted by the group’s typical project work collaborated to identify specific characteristics that drove the project execution effort. As an example, based on our previous project work, we identified the need to develop a completely new application and the need to bring up new hardware as work packages that drove significant effort during project execution. We also identified specific customer-driven characteristics such working with a new customer and the percentage of ambiguous requirements as characteristics that required increased effort as part of the overall project work.
Here is a sample of the project characteristics identified as effort drivers for a typical software application development project:
- New application
- New hardware required
- Online component to architecture
- IT enterprise level support required
- New customer
- External customer
- # of other applications affected
- % of ambiguous requirements
- % of code reuse possible
- Full regression testing
- Changes to core functionality
- # of outstanding tickets/bugs to be fixed
As you can see from the list above it’s possible to identify high-level project characteristics that influence project complexity and drive effort to execute. This identification was done through interviewing the SMEs and team brainstorming and turned out to be the easiest part of building the complexity model.
Developing The Relative Scale
Developing the relative complexity scale was much more difficult to do and required several steps. At this stage the list of characteristics was quite long but through the process of developing the relative scale the list was pared down to a subset of important characteristics responsible for driving effort.
Step 1: Group the Characteristics into Logical Buckets
After the brainstorming and interviews with the SMEs, the identified characteristics were bucketed into high-level groups such as General, Code, Validation, and Release. This grouping is arbitrary and we found that creating groups based on functional areas worked well. The groups and which characteristics fell into these groups was determined by the team and were adjusted as needed to build the model.
Step 2: Assign a Maximum Score for Each Group of Characteristics
Determine the maximum score possible for the model, that is, if a project is of the highest complexity, what should it’s score be? Again, this is a relative score so you can use any number you like, but for simplicity it is recommended that the maximum score be set to 100 points. Now determine the maximum score for each group of characteristics. To do this consider how much effort each group of characteristics contributes to the total project effort. For instance, for our model we knew from historical data that code development plus the validation of the code represented the largest drivers of effort for the project, so we assigned 25 points to each group. You can interpret this as: the code development work contributes 25% of the effort, and the code validation work contributes 25%, therefore 50% of the effort to complete the project comes from developing and validating the code.
At this point if a large number of groups have been identified, it is recommended that like groups be combined to up-level the analysis. Remember, the team will use the model at the scoping stage so keep in mind what type of information will likely be available during the scoping activity when trying to up-level the groups. You may also increase the maximum score possible for the model so that each group has a sufficient amount of maximum points to be distributed in the next step. It is helpful to stick to 100 maximum points, as it helps the team think through the model; however, this is an arbitrary scale so it is perfectly okay to adjust it if it makes building the model easier.
Step 3: Define High/Medium/Low
For each group, the team reviewed the list of characteristics and defined/quantified what aspects of the particular characteristic represented a high complexity project, a medium complexity project, and a low complexity project. It should be noted that these definitions are completely subjective and are based on the team and SME’s experience. Since the end result is an arbitrary scale, this subjectivity does not matter.
Exhibit 1 – Complexity criteria with defined High/Medium/Low states
Step 4: Assign the Points
Assign points to each characteristic and each complexity level (High, Medium, Low). For each group, determine the maximum points possible for each characteristic.
Exhibit 2 – Complexity criteria with defined High/Medium/Low states, group maximum score, and point distribution
In this example, the maximum score for the Validation/QA group is 25 points. These 25 points are then distributed across the five characteristics so as to represent their relative contribution to the total validation/QA effort. For instance projects where large order validation is required take more effort and therefore warrants a higher maximum score of five. Remember, it doesn’t matter how much more effort, only that the increased effort is material and in line with a high complexity project.
After the maximum score for each characteristic has been assigned, walk through each characteristic and distribute points as makes sense. In the example above, if full regression testing is required, then the maximum of 5 points is awarded; whereas, if only partial regression testing is needed, 3 points are counted. Note that some characteristics are Yes/No questions and the score is either the maximum or zero.
Note: if you do not have enough points to spread across each characteristic or the High, Medium, Low complexity levels, then you need to reduce the number of characteristics in the group by combining similar items to up-level the analysis. Alternately, you can increase the maximum score for the entire model and allocate more points to the group. Remember that this is a relative scale so exactly what the final score will be is less important than the fact that the model be applied equally to all projects for evaluation.
Calculating the Project Complexity Score
Once each characteristic, for each group, has a points distribution assigned, calculating the total complexity score is relatively simple. First, configure your spreadsheet to have a “Complexity Score” column where the user of the model will enter a score, aka number of points, for each characteristic based on the particular project to be evaluated. The complexity score for each group of complexity drivers is then the sum of the individual characteristic scores. The total complexity score is the sum of the group scores. Set up your spreadsheet accordingly and begin using the model.
Using the Model
After some trial and error, we determined that it is helpful to convert the information developed in Step 4 above into natural language for the user. To do this, we simply added a column to the spreadsheet. (See Exhibit 3 below.) While evaluating a new project or opportunity, the table of data developed in Step 4 is hidden so that the evaluator only needs to comprehend the natural language equivalent to determine the score for each characteristic.
Exhibit 3 – Complexity scoring range expressed in natural language
To use the model to calculate a relative complexity score for a new project or opportunity, the project or scoping team walks through each characteristic and assigns an appropriate score. The total sum of these individual scores is then calculated for the total complexity score for that new project or opportunity.
A Few Words of Caution
While building this model is fairly straightforward, there are a few pitfalls to watch out for. First, it is common to start with a rather long list of project characteristics. This is fine as the list will be further refined as the model is developed. At that point in the process, the objective is to gather a solid list of all characteristics that drive significant effort. Further, care should be taken to word the project characteristics in such a way that you can determine the scoring based on the knowledge that is available, or known, at the time the scoping activity takes place. For instance, at the scoping stage the team may not know how many hours of regression testing will be required; however, they should be able to predict whether full or partial regression testing will be needed.
Second, it’s not uncommon for the team developing the model to struggle through Step 2, defining what constitutes the High, Medium, and Low state for each characteristic. The goal is to quantify the state in such a way that a clear answer can be provided. Keep in mind that this is an “order of magnitude” analysis and an exact answer may not be possible at the time of scoping so word the characteristic and the High/Medium/Low scale accordingly.
Using Historical Data to Improve the Model
The model as discussed so far allows the user to calculate a relative score that represents the complexity of implementation. It does not, by itself, predict the resource levels needed to execute nor does it predict the expected project duration. To obtain these two valuable pieces of data, you need to turn to historical project performance data. First, evaluate previous projects with the complexity model you have developed. Note that it’s not uncommon to uncover gaps in the model and you may need to tweak the scoring or definitions of High/Medium/Low from Step 2 above. Once you are satisfied that the model accurately predicts the complexity of the project under evaluation, freeze the model.
To correlate the total complexity score with resource requirements and project duration, use the model to score a number of past projects where you have the resource and duration data available. Just how many past projects you need to evaluate is up to you, but you will want enough data to represent the distribution of low to high complexity projects. As you can see from the example below we found that 12 projects’ worth of data was sufficient for our needs. Since the actual evaluation using the model is quick, it is possible to collect complexity scores on the 12 projects within three or four hours. Once you have calculated the complexity score for each project, construct a basic table such as the one below, utilizing your historical data for the number of resources and the actual project duration.
Exhibit 4 - Complexity score correlated to project duration, effort, and resources
Using the data in this type of table allows the team to quickly predict how long it will take to execute a potential project and how many resources will be needed. As you can see above our data indicates that a complexity score of 20 and less is low complexity, 21–40 are medium, 42 and greater are high complexity. Further, as more projects are evaluated with the complexity model and then completed, the complexity scores, actual total effort, and the actual durations can be added to the table above to improve the model’s predictability over time.
How the Model Performs in the Wild
Our teams at Intel have been utilizing such complexity models since 2012 and we have realized several benefits. Correlating the complexity score with historical data has allowed us to improve our project pipeline management in a number of ways. First we have improved the accuracy of our forecasts for expected project duration, since we can now directly connect the project duration to the characteristics that drive effort. Further, since the model is so quick and easy to use, the scoping team can model several complex “what if” scenarios in an hour, identifying the project scope that best fits the available resources and timeframe. In those circumstances where a commitment to execute is needed in a very short timeframe, use of the complexity model greatly increases the accuracy and confidence level of the commit. An unanticipated benefit of using the model is an increase in the team’s overall understanding and buy-in for new projects. Scoring a new project forces a deeper understanding of what it will take to execute, by requiring careful consideration of the major complexity driving characteristics in a systematic manner.
Limitations of the Model and Suggested Improvements
As expected from such a basic data model, there are limitations, but we believe the limitations in no way detract from the usefulness of this tool. First, since the model is constructed of complexity drivers that the SMEs know and experience commonly, the model struggles to accurately predict the complexity for projects with radically different or novel complexity drivers. Second, what was once a challenging work package to implement becomes commonplace and well understood as teams gain experience and exposure to specific complexity drivers. Re-use of designs also reduces the effort to execute formerly complex features. Therefore the model must be updated periodically to incorporate new drivers of effort/complexity and downgrade the complexity score for characteristics that have become well understood and easy to implement.
While the model is extremely useful as is, there are some possible improvements that suggest themselves. It seems reasonable that the model could be expanded to a program level, either by creating a model based on complexity drivers at the program level or by aggregating the complexity scores from the sub-projects that make up the program. In the first case, a program level model, it could prove difficult to gather meaningful historical data to validate the model and leverage the complexity score for predictive modeling. In the case of aggregated project complexity scores, care should be taken to avoid “double dipping” when accounting for interdependencies between multiple projects and to include the effort to integrate multiple projects into a cohesive whole. Careful thought should be given to scenarios where many projects are dependent upon each other and just how that interdependence is expected to drive complexity at both the project and the program level.
Last, we believe that this complexity modeling technique could be employed with projects that follow an agile methodology. Since the calculation of the complexity score is done at a high level, it does not depend on the specifics of the product backlog. Therefore it seems logical that it would be possible to calculate a complexity score early in the project even though the actual sprint backlog and stories are unknown. Correlation of the complexity score with the total number of sprints needed to complete the project would forecast total project duration for a well-established agile team with a consistent and stable velocity. If the story points for an entire project were correlated with the overall complexity score, this would allow for high level “what if” modeling of resources even before the agile team started their first sprint. No work in this space has been done to date, but the possibilities as an early forecasting tool warrant further research.
Today project work is increasingly more complex and ambiguous. The complexity model discussed in this paper allows project scoping teams to quickly and accurately predict resource requirements and expected project duration. It enables rapid “what if” scenario modeling to allow teams to maximize the scope of work given a set resource constraint and a tight timeline. The model is straightforward to build, utilizes the expertise of SMEs, leverages historical project performance data, and most importantly is very quick and easy to use. Today, our team can scope a large, complex project, including “what if” scenario modeling in just an hour. We not only generate a high confidence estimate for resourcing and overall duration, we have a much clearer and deeper understanding of the new opportunity. In the past, this work was typically done by a handful of SMEs in a random manner with mixed results and limited understanding of the work transmitted to the rest of the team. We have found this tool to be powerful and integral to our pipeline management and are pleased to share this modeling technique with our PMI colleagues.
Appendix A: Software Project Complexity Model
Appendix B:IT Infrastructure Project Complexity Model
© 2014, Melanie McBride
Originally published as a part of the 2014 PMI Global Congress Proceedings – Phoenxi, AZ, USA