Beyond earned value management
do you assess a project's team trust index?
Head of Project Management Office (PMO) and Bid Management, adesso AG (Munich, Germany)
Trust is of utter importance for a project. If the project team does not believe “in the project,” the project manager—and with him, the project and the entire team—is likely to fail.
The larger a project, the larger the team, and the more complex it is for the project manager to gain and keep control. A “team barometer,” as introduced in this paper, delivers a snapshot of the team's trust, both in the project's success and its manager, fixed in the “team trust index.” The index is derived from the team member's regular evaluations of a few simple statements about the project situation. It can be used either to validate the findings of earned value management (EVM) and other key performance indicators (KPIs) or as a stand-alone early warning indicator. This paper demonstrates how to join a team barometer with earned value management based on real-life experiences from a 1,000+ employee IT consultancy in Germany.
Earned Value Management (EVM) as a Key Controlling Element in Project Management
Why You Should Measure Trust in Your Project
The company used as an example in this paper is a real German IT consultancy with 1,000+ employees. To better refer to it, it is called ABC Ltd. throughout this paper, although this is not its real name. ABC Ltd. is delivering more than 50 fixed-price projects per year. Most of them range from €0.5 to 1 million just for the workforce. Those fixed price projects have a great advantage for the buying side: The buyer gets a fixed scope with a fixed quality delivered on-time at a fixed price. The risk is transferred to the seller who should better understand the project's pitfalls and slingshots than the buyer. However, in real-life, a fixed-price project has countless unknowns that need to be managed by the seller. Time and cost projections are forecasts that might or might not materialize. M. Sneed depicted the Devil's Square (1987) as an interaction of time, quality, cost, and scope (“Quantity” dimension) as shown in Exhibit 1, in which the four factors are in an antagonistic relationship. As the productivity of the selling party is limited and might not be able to satisfy all needs, trade-offs have to be made. For example, doing more work in a higher quality will result in higher cost and a longer development time (Minich, Harriehausen-Muehlbauer, & Wentzel, 2009, p. 2).
Exhibit 1 – Sneed's devil's square (Eichberg, 2014, p. 13).
In other words: If not all of the factors do perfectly match, the selling party is realizing a loss—or a profit, depending on the shifting in the square. As software development margins are already tight in Europe and the suppliers compete in an open market, gaining profit from a fixed-price project became more and more difficult throughout the last years. Not bidding for fixed-price projects is not an option, as customers repeatedly request that type of contract to limit their risk.
Rigid project management processes can contribute to a better understanding of the project's risks and chances. Based on PMI's A Guide to the Project Management Body of Knowledge (PMBOK® Guide) – Third Edition (PMI, 2004), the supplier developed a pragmatic approach to deliver software projects. In 2010, it published this new way in a book titled Pragmatic IT Project Management - PITPM (Spitczok von Brisinski & Vollmer, 2010) and thus made the processes publicly available. Since then, the model is being reused by a lot of companies and freelancing project managers. The core concept mainly relates to controlling of small units: The big picture is derived bottom-up and not top-down. The most important tools for project management in this environment are:
- A project management plan that precisely describes the project's entire process landscape. This proves to be difficult for most project managers, as lots of processes are not clear right from the start. However, the project management plan is a key driver for risk mitigation.
- A project plan or the Gantt chart. Planning on work package level is prerequisite for estimating the “time factor.” The plan must only change if a change request has been authorized. The work packages shall match 1:1 to those used in earned value (as described later in this article).
- Weekly status reports: Actual situation, changes in risks, current actions, and outlook are the four dimensions the project manager has to describe in bullet points week by week on one PowerPoint slide. The dimensions are supported by easy-to-read indicators (traffic lights) to quickly judge the current situation.
- Risk assessment: The risks are regularly investigated by the risk team. The team is led by the risk manager and is responsible for tracking both risks and effectiveness of counter-measures.
- Earned value report: The PMBOK® Guide' s process “7.4 Control Costs” (PMI, 2013) describes the concept of earned value management (EVM) as an important source of KPIs. While EVM's actual cost (AC) can usually be derived easily, and budget at completion (BAC) is fixed, the ascertainment of the estimate to complete (ETC) can be complex. It requires thought-through planning right from the project start and a project manager who is capable to scrutinize the project team's estimates. From week to week, the project manager is responsible for the “to complete” estimation. Taking the “controlling of small units” seriously, he or she is not guessing the “to complete” for the entire project, but together with each single team member for his or her work packages. Over all packages, this totals to the project's ETC. Adding the project's AC to its ETC results in the estimate at completion (EAC), which represents the actual estimated total cost of the project. EVM knows of numerous additional factors, but AC, ETC, and resulting EAC are the most important components in this environment.
At ABC Ltd., EVM is the key concept from project calculation over planning to execution. Project managers get an “EVM boot camp” when they start, teaching them how to deploy the concept for their and their project's purposes.
This toolset is not bulletproof, but it works quite well for most of the projects. Although rigid planning and project controlling became a bit old-fashioned in our current agile era, all the tools effectively contribute to risk reduction. However, projects sometimes struggle to keep up with the ETCs they reported. This is a pattern that is naturally more common to larger projects than to small ones, as uncertainty is much higher in high-volume projects. Decomposition, as praised in the The PMBOK® Guide – Fifth Edition, is without question the most powerful method to understand the scope, define work packages, and come to a reliable estimate. What sounds simple in theory often turns out to be very difficult in practice. Most long-tail projects cannot be thought through at the first day from start to end. And there are still project managers who are either not familiar with the concept, do not see the advantage of EVM and the other concepts, or who simply do not adhere to the transparency created through those processes. Given that, EVM is a good idea—but unfortunately does not guarantee that a project becomes a success.
The PMO regularly discovers projects that fail to their own expectations, although the figures are estimated frequently on a weekly basis. Although the PMO investigated each project that failed to deliver according to its own earned value analysis, it did not discover a pattern that led to the wrong estimations. The turning point came when a project manager of a €1 million + project reported a loss in cost performance index (CPI) of 0.18 during seven days from 1.03 to 0.85, with no change in the project's environment. This was a real shock—was the PMO just lucky that the other projects were successful? Or did the project manager lie during the last weeks and did not apply the “controlling of small units” properly?
When talking to a software engineer of the failed project about what happened, he flushed and whispered:
“I could have told you much earlier. We lost track a few months ago.”
“Why didn't you tell?”
“I thought you all knew. It was so obvious.”
Windbag or whistle-blower? Whatever—this was a starting point. PMO defined a system that lets each team member of a project evaluate project performance, team spirit, and project manager leader skills from his or her own perspective: 10 simple questions, asked anonymously every week, taking less than a minute to answer. A very simple tool of PMBOK® Guide – Fifth Edition process “4.4 Monitor and Control” (PMI, 2013). ABC Ltd. named this a “project trust measurement solution,” continuously delivering a set of figures about the project team's trust in their own success. Those figures may verify—or falsify—KPIs like EVM, as explained in the subsequent sections.
Simple but Efficient Tools to Measure Trust
The approach to measure trust in a project may have different factors of influence. “Trust,” as used in this context, at least covers the following:
- The project team members trust their teammates and their will to both cooperate and deliver their work packages on time.
- The team members trust the project manager, his way of managing the buying side's expectations, and his understanding of how to run a team and lead it to success.
- The team members trust that they can deliver the project in time, in budget, and in quality.
More aspects of “team trust” can be added.
A team is in constant development, which also effects the trust in the team. The project manager and the entire team shall be aware of their actual team development phase. Several models exist to express the current phase; one of the most respected is the Tuckman ladder with its five phases (Tuckman & Jensen, 2013), represented in Exhibit 2 in the form of a “team clock,” with “12” being project start and end:
Exhibit 2– Team clock (following (PMI, 2013, p. 276) and (Tiemeyer, 2010)).
The team phases influence the evaluation of trust in a project. While in the initial two phases, Forming and Norming team members might not feel comfortable in the team and need to “stake their claims” in the Norming phase, trust in each other's skills and capabilities becomes vital for project success. On the one hand, evaluations derived with the team barometer can help to determine the team phase; on the other hand, the team phase might influence the team member's evaluation behavior. Thus, the project manager shall know at least roughly in which phase his or her team actually works.
Measuring trust shall be a permanent process for each project manager to make results comparable over a timeline. This also requires to decompose the term “trust” into measurable and sizeable units to avoid that impressions might be interpreted differently from one evaluation to the next. The evaluation shall be done periodically (e.g. weekly or monthly) and therefore must not take too much time for each team member. Plus, the “measurable and sizeable units” shall be easy to understand for all team members. For this purpose, the company came up with a set of statements each project team member shall evaluate periodically. The statements are the same in each project, plus the project manager can add statements of his choice. A statement shall be phrased in a way that a team member can quickly make his or her decision. Current statements are:
- The results I deliver have high quality.
Explanation: Are you able to deliver to your expectation? Is the project environment shaped in a way that you can deliver what others expect from you?
- Communications within our team work very well.
Right after project start, communication is often poor. If it fails at a later project phase, this is a call to action for the project manager.
- Communications with other sub-projects work very well.
Often a problem in large-scale projects. This statement can be divided into one for each sub-project to understand which need further investigation.
- Our team delivers what our customer expects.
It sometimes turns out that project team members do not know what the customer expects, or the customer has not yet expressed his needs in detail. If so, the project has a severe issue. It might also turn out that the project works in accordance with the scope statement, but the customer expects something different to that.
- My role in this project fits to my skills.
This statement is required to understand if all project team members have “arrived” in the team. The evaluation can be two-fold: If skills and role do not match, the member can either be unable to cope with the demand, or he or she is bored.
Both need the project manager's attention.
- We will close our project in time.
This is to understand if the team member trusts the project timeline.
- We will close our project in budget.
This is to understand if the team member expects that the budget allocated to the project is sufficient. If work packages take longer than expected or turn out to be technically more complex than anticipated, this can be an early warning indicator.
- The project manager masters the project.
This is to understand if the team members accept the project manager, his role, and his duties, and if they trust him to lead the project to a success.
The handling of the evaluation can range from simple free-form discussions periodically held in the team to electronic forms provided to each team member. Common implementations to let the team periodically evaluate trust are:
- Regular meetings
The project manager leads the discussion about the statements listed above in a free form. Team members contribute whenever they have an opinion to share. This form can be very inviting to extraverted people, while incommunicative ones might prefer not to participate. This is the weakest form of the team barometer, as evaluations are not fixed in numbers and the development of selected figures cannot be tracked. However, it might work well for teams that are in their Norming phase or beyond.
- Team barometer white board
A basis for the regular team barometer meeting can be the team barometer white board. This is a space dedicated solely for the team barometer evaluations in the team room where all team members have been co-located. Before a meeting, each team member evaluates all statements with his or her personal color on the white board. A range of 0 to 2 proved to be helpful for this approach. 0 means “I fully disagree,” 0.5 means “I mostly disagree,” 1 is “undecided,” 1.5 is “I mostly agree,” and 2 indicates “I fully agree.” This is a very transparent implementation of the team barometer. It encourages a data-based discussion rather than an exchange over opinions and emotions. However, reserved team members might feel offended by the very transparent documentation of the evaluations. They might also tend to provide an evaluation that they believe is “politically correct” to avoid confrontations among team members and the project manager.
- Using simple electronic tools
The statements listed above can easily be entered in freely available survey tools. Project managers could also use plain Microsoft Excel or other spreadsheet software to periodically retrieve the team member's evaluations. The project manager lists all statements in the respective tool and sends around the link to the form or the evaluation sheet. With this method, often even shy team members participate and provide their opinion. A drawback of this approach is that the project manager needs to do some manual work before and after each evaluation. At least in the web survey tools, tracking of the evaluations over time is not possible and requires manual extensions. As suggested above, the project manager shall discuss the team barometer results in a regular meeting.
- Professional team barometer solutions
The market for team barometer solutions is still pretty small. Some companies offer a comprehensive team phase analysis to investigate the entire team with numerous questions. Due to the nature of the product, those solutions are not intended to be repeatedly performed in short cycles. Other platforms fully focus on the team barometer, which allow a set of evaluations that are periodically evaluated by team members. The development of each evaluation can be tracked, team members participate anonymously, and the statements can be evaluated quickly, which allows evaluation in short cycles. (Spitczok, Vollmer & Weber-Schäfer, 2014, p .160)
In all team barometer implementation scenarios previously described, the project manager shall discuss the results with the team members and shall take corrective action whenever needed. Deploying a team barometer without being willing to take corrective action can be counter-productive. Team members usually adapt quickly and might understand their project manager's behavior as an accepted way of treating issues: If you do not talk about an issue, it is nonexistent. Thus, setting up a team barometer implies a great deal of responsibility for the project manager to not abuse the tool.
How to Calculate the Team Trust Index and What to Do with It
The team trust index is the overall result of the team barometer and comprises all evaluations of all statements requested from the team members for one period. Each evaluation has the same weight, regardless of its provider and the statement's nature. The team trust index is the overall average of all evaluations provided over all statements.
Let s be the statement with n being the number of statements in the poll. Let e be the evaluation provided by each participant per statement, and let m be the number of participants per evaluation. Using p for the actual period, the team trust index tp is then calculated as described in Exhibit 3:
Exhibit 3– Formula to calculate the team trust index t for a given period p
An evaluation ranges from 0 to 2 with 1 being the neutral mean. The idea behind that range is to use the index in the same way as the schedule performance index (SPI) and the CPI: Values above 1.0 allow the conclusion that the team trusts in the project, values below indicate room for improvement.
The following graph shows the development of the single statements of a team barometer as introduced earlier. The thick black line indicates the team trust index. The bottom blue line shows the team's evaluation of the statement, “We will close the project in budget,” which is close to zero. It's obvious that the project has a budget issue.
Exhibit 4 – Results from a bi-weekly performed team barometer.
As stated earlier, using the team barometer without working with the results might be counter-productive. Project managers must read the graph to fully understand how the team thinks about the project. As with the indices in EVM, values below 1.0 indicate performance below expectation, while values larger than 1.0 mark a performance better than expected. Taken the graph from Exhibit 4, three statements continuously earn high values and thus mark strong areas of the project:
- Our team delivers what the customer expects.
- Communications with other sub-projects work very well.
- My role in this project fits to my skills.
One further statement alternates around 1.0 but still moves in an acceptable range:
4. The results I deliver have high quality.
Still in an acceptable though lower range, but with alarming amplitudes:
5. Communications within our team works very well.
And finally, three statements that shall put the project manager on alert:
6. We will close our project in time.
7. The project manager masters the project.
8. We will close our project in budget.
The project manager can use the statements clearly above 1.0 to strengthen the project and the team. The team seems to be focused, which is indicated by high evaluations for (1) “Our team delivers what the customer expects” and (3) “My role in this project fits to my skills.” A potential drawback is that this dedication to customer satisfaction might jeopardize the project's time and budget baselines as indicated by poor results in (6) “We will close our project in time” and (8) “We will close our project in budget.” This potential correlation needs further investigation by the project manager, who himself is put into question by the team: Statement (7) “The project manager masters the project” delivers the poorest possible results. The team does not trust the project manager at all, and whatever led to that disastrous result needs to be investigated. This is a task that potentially needs to be performed by the project manager's line manager to avoid a conflict of interests.
What the graph above does not express is the standard deviation that comes with each evaluation. Taking just the average result per statement might be misleading. For a given statement, the evaluations as sampled in Exhibit 5 deliver the same overall result but have a diverse meaning:
Exhibit 5 – For a given statement, Team 1 and Team 2 deliver the same result with very different evaluations. This is indicated by the standard deviation.
Both teams have the same average result of 0.94 for that statement, but it is derived from diverse ratings. The ratings of Team 1 range around 1.0 without any extreme values and show only a slight deviation. Team 2 does not deliver a consistent result: While half of the team members fully agree to the statement, the remainder completely disagrees—with the same average result. Such a diverse picture needs further investigation and may also be an indicator for the team still being in the Storming phase.
How the Team Trust Index Can Contribute to Project Management Quality
The team barometer is still in development. There is no scientific proof that it effectively contributes to more successful projects. However, the results from the first 15-20 projects are promising. Project success put aside, in all projects monitored with the team barometer, the communications within the project improved significantly. If introduced properly by the project manager, it contributes to a friendly environment for communication. It has also been used as a communication-enabler in wrecked projects as suggested by Dannenhauer, Koerting, and Merkwitza (2013, p. 161) with great success.
The team barometer will only provide the project manager with a snapshot of the current situation. It will also show the development of each statement and relations between them. It will not provide the project manager with a detailed analysis of the team's strengths and weaknesses, and it will not suggest any measures.
Understanding the limitations of EVM – Why EVM Results Shall Be Cross-Checked with the Team Trust Index
As described at the beginning of this paper, EVM is one of the most important tools for project controlling. ETC and EAC are the most important components of EVM. Both values shall not be used stand-alone, regardless if ETC is estimated bottom-up or top-down. Following the dictum of “controlling of small units,” the project manager will achieve more reliable estimates if performed bottom-up on work package level. This is more costly, but if introduced right from project start, it is more a matter of discipline than cost. Project team members usually quickly acquaint themselves with regular estimations on a work package level.
Regardless how the EAC is retrieved, it needs to be tested against other factors to get a certain level of reliance. If estimated bottom-up, project managers might use the EAC forecast for ETC work performed at the present CPI to test the estimation. If the gap between both figures is too large (e.g. an 8% deviation is accepted), the project manager needs to investigate potential causes. If the project manager cannot identify feasible causes, he or she needs to revise the estimation. If the ETC is estimated top-down on a project level, it is difficult to find other values to test the estimation.
In such cases, line managers may find themselves exposed to sudden ETC bursts in their projects. An ETC burst describes a massive increase in ETC without a change in the project scope. Such bursts may have different causes:
- “Controlling of small units” has not been applied at all or has not been applied properly
- Faulty estimations from team members
- Unexpected extensive defects in the software (which often have their cause in faulty estimations, though)
- External reasons, like sick leave or changes in staff
Exhibit 6 demonstrates how an ETC burst is expressed in an EVM chart:
Exhibit 6 – The two red arrows mark the ETC bursts. This example is taken from a real project.
The two red arrows mark an “ETC burst”: An unexpected massive increase in ETC. An ETC burst can have many causes and usually wreaks havoc for the project and might even—depending on size—affect the entire company. The graph in Exhibit 6 is taken from a real-life project, an implementation of a content management system for a client from the automotive industry.
In most cases, the project manager is responsible for the burst. For the line manager to be not at the mercy of his project manager, he or she should insist on testing the earned value results against other factors. The team barometer can be used quite easily to test the estimations, at least on a high level. In the projects where the barometer has been used already, most team members have a realistic view on the overall project situation. They express their assessment in the team barometer evaluations.
In the example shown in Exhibit 6, ABC Ltd. introduced the team barometer right after the first ETC burst. The graph in Exhibit 7 stems from the same project as the first graph and shows the project's CPI (brown line, left axis) and its cost variance (CV, blue line, right axis). After the first ETC burst (red arrow 1), ABC Ltd. introduced the team barometer. The thick black line shows the team barometer's team trust index. The second burst was indicated by that index already some weeks earlier.
Exhibit 7 – The same project as in the previous exhibit, now supplemented with the overall team trust index (black line).
Conclusion: Strengthen EVM with the Team Trust Index
The team barometer and its team trust index can be used as an early warning indicator. In this scenario, it has been deployed to test the earned value results. Most team members have a good sense to judge the overall project situation and can express their notion quite well in the barometer. Nevertheless, although the tool provides indicators, it does not exactly identify potential issues and problems. Furthermore, it does not provide a solution for a difficult situation. All it does is indicate that there might be an issue. The project manager then needs to investigate the causes jointly with the project team. Asking the right questions as outstandingly explained by Brown and Keeley (2014) will help the team succeed.
The team barometer can be used stand-alone, and then is a smart tool to get structured feedback on a regular basis. It is even more powerful if used jointly with the results from EVM. The team trust index or, even more detailed, the statements about time and budget can be used to test the results of the EVM. Line managers can use the barometer to quickly assess their project portfolio, either to run it against EVM results or just as a standalone tool.
Remember: The team barometer will not solve the project manager's team management issues—but it will alert him or her if there is one.
Brown, M. N. & Keeley S. M. (2014). Asking the right questions: A guide to critical thinking. New York, NY: Pearson Education Inc.
Dannenhauer, R., Koerting, T., & Merkwitza, M. (2013). Turn around. Wenn projekte kopfstehen und klassisches projektmanagement versagt. Frankfurt: TURN AROUND Thinktank GmbH.
Eichberg, M. (2014). Introduction to software engineering. Retrieved on February, 14, 2014 from http://www.stg.tudarmstadt.de/media/st/teaching/courses/ws2009/material_eise/20091028b_softwareprojectmanagement.pdf
Minich, M., Harriehausen-Muehlbauer, B., & Wentzel, C. (2009). Software industrialization in systems integration. Retrieved on January 3, 2014, from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.193.4656&rep=rep1&type=pdf
Project Management Institute. (2004). A guide to the project management body of knowledge (PMBOK® guide) – Third edition. Newtown Square, PA: Author.
Project Management Institute. (2013). A Guide to the project management body of knowledge (PMBOK® guide) – Fifth edition. Newtown Square, PA: Author.
Sneed, H. M. (1987). Software management. Cologne, Germany: Müller GmbH.
Spitczok von Brisinski, N. & Vollmer, G. (2010). Pragmatisches IT-projektmanagement – Softwareentwicklungsprojekte auf basis des PMBOK® Guide führen. 1. Auflage. Heidelberg: dpunkt-Verlag.
Spitczok von Brisinski, N., Vollmer, G. & Weber-Schäfer, U. (2014). Pragmatisches IT-projektmanagement – Softwareentwicklungsprojekte auf basis des PMBOK® Guide führen. 2. Auflage. Heidelberg: dpunkt-Verlag.
Teampoll. (2013). Are you assessing the performance of your team? Retrieved on August 15, 2013 from http://teampoll.de/
Tiemeyer, E. (2010). Handbuch IT-projektmanagement. München, Germany: Carl Hanser Verlag.
Tuckmann, B. W. & Jensen, M. A. C. (2013). Stages of small group development revisited. Retrieved on September 23, 2013 from http://www.freewebs.com/group-management/BruceTuckman%281%29.pdf
This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI or any listed author.
© 2014, Niklas Spitczok von Brisinski
Originally published as a part of the 2014 PMI Global Congress Proceedings – Dubai, UAE