Raising the bar at Intel
a project retrospectives case study
This case study is designed to share real-life project retrospective data from Intel Corporation where project managers have been asked to gather key learnings from programs. Most teams have zero travel budget, complex cross-organizational cultural differences, and a high level of resistance to change. We discuss how a retrospective is different from other review practices such as post project reviews and post mortems with the four phases of the retrospective process outlined in detail; Plan, Do, Check, Act (PDCA). Through the presentation of the case study, the critical role of the facilitator is described in such a way that the participants can design and deliver an effective retrospective when they get back to their organizations.
In 2002, I read the book, Good to Great: Why Some Companies Make the Leap…And Others Don’t by Jim Collins (2001). As I was reading the book I kept thinking…I wonder if retrospectives will play a part in separating those companies that are good from those that make the leap to greatness. Finally, on page 72 I found it! “When you turn over rocks and look at all the squiggly things underneath, you can either put the rock down, or you can say, ‘My job is to turn over rocks and look at the squiggly things,’ even if what you see can scare the hell out of you.” This is a quote from Fred Purdue, a Pitney Bowes executive who spends two hours at their annual management meeting discussing all the “scary squiggly things” they need to talk about to continue to be a successful company. Their description of the annual meeting sounds a lot like a retrospective.
Siemens has been using retrospectives for many years and seeing terrific results. Standard Insurance uses them and is finding them valuable. The State of Washington is just beginning to apply the practice of retrospectives and is excited about the possibilities. Since 2002 I have had the pleasure of incorporating retrospectives into the work that I do at Intel. I use this project management ritual to help teams improve productivity and increase the quality of our products. The results have been surprising. A seasoned project manager estimated an effective four-hour retrospective could add up to releasing his software project FOUR MONTHS earlier! The ratio of one hour equals one month time savings is pretty exciting.
Many companies support a formal process to allow employees to communicate what they feel the company is doing well and what it needs to do differently. Again from, Good to Great: “shoving rocks with squiggly things in their faces, and saying, look! You’d better pay attention to this. In the book Project Retrospectives: A Handbook for Team Reviews Norm Kerth (2001) describes a process to effectively facilitate retrospectives using specialized techniques to create an environment based on safety and trust. His methodology encourages the team to spend 2% of the total project time (typically three days) to uncover new ways of working together more effectively. The objective is for the team to use this information to increase the capability of the team by:
- Sharing perspectives to understand what worked well in the project so the team can reinforce it
- Identifying opportunities for improvement and lessons learned so teams can improve future projects
- Making specific recommendations for changes
- Discussing what the team wants to do differently
- Helping the team see ways to work together more effectively.
Kerth’s approach is very simple: “A retrospective is an opportunity for the participants to learn how to improve. The focus is on learning—not fault finding.” This approach is the foundation of the practice followed at Intel. For a retrospective to be effective and successful it must be done in the context that no matter what is discovered and discussed, it is in the spirit of improvement. Kerth shares in his book a way for facilitators to create a safe environment where the root cause of an issue can be discussed and solved without fear of management using this as an opportunity to reprimand or punish. A key point in his book is the Prime Directive: Regardless of what we discover, we must understand and truly believe that everyone did the best job he or she could, given these four things:
- What was known at the time
- His or her skills and abilities
- The resources available to them
- The situation at hand.
The prime directive sets the tone for the retrospective to be an opportunity to improve, not finger point or blame. Kerth unveils a common sense approach of using different exercises to reveal what really happened during the life of the project. In 2002, I was lucky to meet Norm Kerth at the Rose City SPIN (Software Process Improvement Network) in Portland, Oregon. To my surprise he lives in Portland and was willing to engage in off-line conversations about his methodology and techniques. After reading his book we had several telephone conversations and face-to-face meetings to brainstorm how his concepts could be modified to fit into Intel’s culture. Below is what has been implemented as the standard framework at Intel. We have adopted a four-step approach following the Plan, Do, Check, Act (PDCA) model:
- Plan—Get ready by spending time to collect project data using a survey/questionnaire and/or interviews and draw on this data to set the stage.
- Do—Look at the past by recreating what happened on the project from all perspectives to harvest the key nuggets.
- Check—Review action plans and spend time to broadly communicate outcomes to senior leaders.
- Act—Close the loop by ensuring action plans are implemented, key learnings are documented in the corporate centralized repository for others to also learn from, and the next retrospective is being planned.
In 1999, I joined Intel as a member of a small team within the Corporate Platform Office (CPO) responsible for driving best practices within our product development teams to reduce rework, increase productivity, and ultimately improve the quality of our products. The focus of my work is to increase the adoption of the retrospective practice as a standard framework to discuss and learn what issues the team has been grappling with so we can reduce repetitive mistakes, increase re-use of proven solutions, and ultimately deliver innovative solutions to business partners across Intel that add value and meet their business needs. We collaborate with product development teams early in their product lifecycle to discover, design, develop, deploy, and evolve business management solutions that improve business results.
What is our strategy?
- Identify root causes and address fundamental business issues
- Research and develop a pipeline of services and solutions needed to deliver value to the business
- Role model collaboration, leadership, and accountably
- Deliver measureable business results through business unit engagements.
Over the past several years the organizational learning and retrospective practice has been gaining momentum as a way to improve our existing product development “post mortem” process and provide a way to apply the findings back into subsequent projects as a method of continuous process improvement. A review of the current “post mortem” practice uncovered:
- Project managers were facilitating their own post mortem sessions that didn’t allow them to share their perspective of the project
- No defined process that kept the discussions constructive
- Post mortems were done ad hoc with no streamlined/consistent process
- Meeting held in some cases months after the end of the project, or not at all
- Lists of issues collected with no action plans associated with solving them
- Information recorded was many times not useful to other projects—for example, not generic enough, not process relevant
- No consistent format of how to document findings
- Meeting length was between 1-2 hours
- No central repository to share information across teams
- Key learnings’ out of post mortem were not easily retrieved to address a specific deliverables
- No closed loop process in place to require key learnings to be considered for subsequent projects.
The approach we took to improving the process:
- Establish a consistent process for gathering, sharing, capturing and documenting lessons learned
- Plan and hold multiple project retrospectives at strategic milestones along the lifecycle and within two to six weeks after the end of the project
- Use a neutral, skilled facilitator to ensure consistent use of the retrospective framework
- Create action plans for the top issues the team wanted to change.
- Share Best Known Methods (BKMs) via a centralized repository using a customized SharePoint site to ensure findings are available and shared across project managers
- Communicate regularly at special forums, staff meetings, and management review committee meetings key findings and action plans to ensure visibility of the changes and support from senior leaders.
This paper is a case study, designed to share real-life learnings from many types of Intel projects. We interviewed project managers all over the world who shared with us what they liked about the retrospective process and what they found valuable. We provide a summary of the process used and results obtained over the past eight years while implementing a standard retrospective practice. The scenario to capture learnings gets multifaceted when teams are geographically dispersed with complex cross-organizational cultural differences, no travel budget, and a high level of resistance to change.
The goals of a retrospective are:
- Understand what is working well so we can reinforce it
- Capture what is not working so we can improve it
- Create action plans for top improvement areas
- Communicate the findings to management.
This case study spans from March 2002 to June 2010. In a little over eight years, approximately 500 retrospectives have been conducted in various internal project and platform teams. Looking at approximately 80 retrospectives, beginning in Q4 of 2009 and projecting out to Q4 in 2010 we believe:
- The majority, 54%, of the retrospectives will be conducted at the end of the project at the Product Release Qualification (PRQ) milestone as part of the standard process supporting the Intel Product Life Cycle
- Approximately 32% during the execution and development of the product
- 14% focused solely on the strategic planning process of exploration, feasibility of the product during requirements gathering and documentation.
Exhibit 1 is a breakdown, by quarter, of actual retrospectives conducted Q4 2009 through Q2 2010 in one organization. We have forecasted out to the end of 2010, based on the anticipated milestones the projects will hit this year. As shown, the retrospective practice is becoming more widely adopted mid-way through the project lifecycle, rather than waiting until the end of the project when learnings can’t be applied real time. At Intel, the projects span from several months to several years, so stopping along the way is imperative so learnings are captured while teams are still together and can reap the benefits of making course corrections. Another advantage to holding multiple retrospectives earlier in the lifecycle is subsequent project managers are invited to attend, listen to the challenges projects ahead of them are facing, so they can use the lessons learned as an early warning system to prevent the same mistakes from happening on their projects.
Exhibit 1. Breakdown of Retrospectives By Project Milestone
Using the PDCA phased approach; during Phase 1, Plan we focus on preparing and planning for the retrospective by collecting project data as pre-work, using a web-based survey tool. We use the feedback to identify key issues that the team should focus on. We feel a survey is helpful when:
- Too many team members to interview
- Geographically dispersed team with many members unable to attend the retrospective meeting due to time zone differences
- Safety and trust level of the team is low
- Cultural and language issues prevent the team from sharing openly.
The pre-work allows us to collect the team’s opinions and perceptions on what and how they did in the project. Their feedback is confidential, however, we know who answered the survey so we can get a response rate and confirm a stratified sample, but we don’t correlate the respondent’s name with the specific comments. We recommend the survey be sent to anyone who has a perspective to share. We have found giving the respondents at least two weeks to answer the survey provides the best response rate. Exhibit 2 shows a summary of 20 random surveys conducted from Q2 2009 through Q2 2010.
Exhibit 2. Summary of 20 Random Surveys Conducted
Facilitated review of the survey data encourages the team to jump into meaningful discussions centered on these five open-ended questions:
- From your perspective, what worked well?
- What did you learn?
- What should we do differently next time?
- What still puzzles you?
- What area or process requires a deep dive discussion or follow up later?
The first survey we conducted in 2002, received a response rate of only 11%. We attribute the low response rate to:
- Team didn’t understand the objective of the survey. Team members were confused because previous “post mortems” had never asked them to answer a survey. They thought they could attend the face-to-face meeting and verbally provide feedback like they always had done. We now have a standard e-mail (Appendix A) that goes out to the survey takers explaining the retrospective process and the survey as prework.
- Not enough time to respond to the survey. We allowed only three days for the team to take the survey and we didn’t send “gentle reminders.” Even in Intel’s disciplined culture, we found we must allow at least two weeks for participants to respond to the survey. We consistently obtain a response rate of over 70%. To get this type of response rate we send MULTIPLE reminders via e-mail and ask project managers to verbally encourage the team members during weekly team meetings. Very rarely, in extreme circumstances (illness, travel, and key stakeholder who can’t log into the network) we extend the time to take the survey.
- Too many survey questions. We asked 78 questions that spanned the entire 18-month project. Today, we ask no more than 10 questions, the standard survey is only five, open-ended questions. We have standardized on three key milestones where we intercept the team and recommend a retrospective (just covering a portion of the project) and found the participation and response rate improved.
- We didn’t control who was asked to participate in the survey. We sent an email to approximately 60 key stakeholders, and added (what we thought was a benign statement) to feel free to forward the e-mail to others on the team they felt would have a perspective to share on the project. We estimate the e-mail was sent to over 260 project stakeholders. We have a strict practice to NOT forward the web survey link so we can control the response rate and ensure we have feedback from critical stakeholders.
We feel the pre-work (web-based survey) provides a consistent process for gathering data about the project and provides a basis of anonymous feedback for the team to learn from.
Once we have gathered and analyzed the survey data, we move into Phase 2, Do. This is where we bring the team together to look at the project by recreating what happened from all perspectives and harvest the key learnings.
In 2002 we insisted on a four-hour, face-to-face meeting within two to six weeks of the project close. Today, most teams don’t have a large travel budget and are too busy to devote a half a day to talk about the project. Many are spread across multiple geographically locations. So we have evolved to where the bulk of our retrospectives are held virtually, using collaboration tools and a conference call, where 100% of the participants are “dialing in.” We don’t want to miss an opportunity to document the team’s wisdom by waiting until funding dollars or other face-to-face meetings are available. Using the collaboration tools and technology, the information about the project (while still relatively fresh in the team’s mind) is now discussed from the comfort of their home office or desk. In every retrospective, we used a neutral, skilled facilitator to ensure consistent use of the retrospective methodology so we can maximize the time and energy a team dedicates to lessons learned. Norm Kerth’s exercises and methodology jump started us into effective activities to summarize the survey data and draw out all the “squiggly things” the team should be discussing. After conducting hundreds of retrospectives, we found the retrospective activities were becoming predictable by our teams, so we employed Esther Derby and Diana Larsen’s (2006) Agile Retrospectives: Making Good Teams Great book to breathe new ideas and approaches to our shorter, more condensed virtual retrospectives.
We found keeping a team on a telephone call for more than two hours was not working very well. Energy levels decreased and brains were too tired to do action planning. Feedback from our teams indicated two 2-hour meetings were preferred. See Appendix C for sample agendas. It takes two hours to sufficiently understand the issues, and another two hours to fully develop plans to change. The goal is to ensure we have committed owners, willing to apply and track the impact of the improvements to their projects by fully documenting and publishing what was acted upon. My wish is at some point we will normalize on allocating 2% of the total project time to reflect back and apply forward key learnings as Norm Kerth suggests. Based on what we found at Intel, the most effective retrospectives are those done more frequently, in a virtual setting, reviewing smaller chunks of project time.
Selecting the right approach (face-to-face or virtual), identifying the right activities (timeline exercise, etc.) and asking the right questions (open-ended vs. closed) has allowed us to uncover a remarkable amount of data about our projects. The goal of every retrospective is to identify the top three to five items the team felt were the most important to continue doing (things that went well), and identify the top one to three “squiggly things” they know needs to change. Exhibit 3 summarizes the top themes and trends from 20 random retrospectives:
Exhibit 3. Top Themes and Trends
In Phase 3, Check we document and review action plans to broadly communicate the retrospective outcomes to management. The purpose of this double-check is to confirm with the team’s senior leaders the issues they identified are supported by them and offer help needed to remove obstacles. We insist all teams prepare at least one Action Plan from their top three to five improvement areas. The Action Plan Template (Appendix B) allows the team to quickly, usually within 90 minutes or less, document the issue and start to formulate a plan to make a change.
Some teams are asking for help to probe deeper into the preliminary action plan work and creating A3’s to further enhance the Check phase. Teams are studying ways to reduce waste and implement a more robust solution to the problem the team has prioritized to go work on. The A3 methodology originated from the Lean approach (www.lean.org) where the team is confined to one 11″ x17″ piece of paper (taken from the European sized paper named an A3). The team uses just one piece of this sized paper to force them to be concise with their proposed solution. Many of the teams at Intel use an A3 approach to dissect problems using a “new perspective,” which helps to identify specific problems from the customer’s point of view. Restating the problem this way helps support and reinforce that we need a deep understanding of the current condition, from the customer’s perspective before we jump into solution space. By defining what needs to happen, who needs to do it, and by when with an expected outcome we can discuss realistic solutions. Built into the process are specific counter measures, with owners and due dates, which enables us to check progress and ask ourselves “is this achieving what we expect?”
Phase 4, Act is the final step where we close the loop by ensuring action plans are implemented, key learnings are documented in our centralized repository for others to also learn from, and the next retrospective is being planned.
How do you know if an action plan is being implemented? Within a few weeks of a retrospective, teams are instructed to present their action plans (and occasionally all the collateral documented from the retrospective) using a PowerPoint template summarizing the team’s learnings or BKM with their division project managers, team leaders, group project managers, or senior management so findings can be incorporated into the upcoming projects to stop repetitive mistakes, re-use solutions, and ultimately improve productivity and product quality. Clear and concise problem statements and all supporting information such as a detailed A3 or survey data is available so management can get a complete picture of the issue identified by the team in the retrospective.
In addition, we schedule quarterly follow up one-on-one meetings (e-mail didn’t work very well) with the action plan owners and ask them five questions:
- What improvements have been implemented and what was the impact?
- What improvements have been started, but not complete yet?
- What improvements haven’t started yet, but still important to the team to implement?
- What improvements were handed off and to whom?
- What improvements did the teams decide were not important and are not working on?
The feedback from the project teams proved very interesting. The teams reported many improvements such as reduced rework because the completed action plans were reported to management shortly (two to four weeks) after the retrospective meant the learnings were visible to a broader audience. Project managers working on solving an issue were able to collaborate with the team with the most forward momentum and reduced redundant efforts, which allowed management to give their full support to making the recommended changes.
Requirements, mentioned in Figure 3, are often a focus area for retrospectives conducted during the early exploration and planning phases of the project lifecycle. One team spent time to minimize requirements volatility by developing detailed Use Cases early in the planning phase to help gather and validate requirements because their old method wasn’t meeting their needs. The impact, the newly formed Use Cases helps the team to drill-down into unclear requirements, resulting in the elimination of “false” features and ultimately improved the product quality and improved productivity of subsequent teams who used the use cases.
Another improvement was increased product quality. One team was able to implement a better Change Control process using existing bug tracking tools based on discussions from the retrospective. The team reported the product quality improved because project scope and risk management was targeted with action plans from their survey discussions and findings.
One team reported they were able to shorten the time to market (TTM) because during the exploration and planning phase of their project they uncovered a single point of communication was needed. The team implemented a website for the project to improve communication and reduce time to market. The team determined it was much easier to find documents and understand where the project was at all times by using the website to post all project collateral. This saved the team time searching for information and allowed more time to actually work on the development of the product.
Unfortunately, several of the teams, when asked what actions were taken from the retrospective responded with “What action plans? Oh, those we filled out in the meeting? Well, I think they are on my desk somewhere!” I assumed the project teams would take the action plans and do something with them. I didn’t think it was necessary for me to keep a copy of the outcomes. A few months later when I queried the teams on what they actually implemented they asked me “What was it we said we would do?” I now keep a copy of all the action plans developed by the team in a shared folder where access is open to all employees and follow up with the team regularly (quarterly) for updates and changes. Because I had not documented every action plan the teams created, I had no idea what had been implemented and could not form conclusions as to the percentage of items identified to changes implemented.
When I reported out the results to the divisional manager, I asked: “Based on the results I’ve shared with you, do you feel the retrospective process we have implemented within your division has saved at least X hours (X=total number of hours invested in retrospectives)? This is based on the improvements implemented as a result of the outcomes of the retrospectives.” The divisional manager answered “I don’t know. Based on your report you have missed several items we implemented such as a validation task force and a tiger team to improve requirements activities.” I learned, don’t assume and take the team’s word for what THEY think the impact of a retrospective has been to their projects. Ask senior leaders too, they probably have a broader perspective.
The overall impact of implementing a standard retrospective practice within Intel is still a work in progress. However, the majority of the project managers we interviewed for this paper stated they were very happy with the intangible outcomes such as increased communication and improved productivity. One project manager estimated the time savings attributed to an effective four-hour retrospective could be up to four months on his current project alone. The ratio of one hour equals one month savings has encouraged our Corporate Platform Office to continue to explore and expand a standard retrospective framework worldwide.
Over the past eight years the goal has always been to establish project retrospectives as a standard practice for continuous process improvement for all projects at Intel by:
- Defining and documenting a consistent retrospective framework
- Sharing key learnings via a centralized repository
- Reporting findings to upper management with specific action plans for improvements with owners and due dates.
In closing, what “squiggly things” did I find that I would do differently?
- Launch an “Apprentice” program to build retrospective expertise so the practice will thrive after the process improvement “guru” moves on. We have developed an internal full-day face-to-face and a four-hour virtual training course resulting in over 125 trained facilitators. The intent is for qualified facilitators to reciprocate facilitating retrospectives for one another. The project managers are encouraged to NOT facilitate their own projects, rather as the senior leader they have an important perspective to share and it is difficult to do that while facilitating the discussions.
- Conduct multiple retrospectives throughout the project life cycle. Most of the projects at Intel projects are 18 months to three years in length. When we interviewed the project managers and asked them what they thought worked well and what they wanted to do differently at the end of the project, the team only remembered what happened during the last few months of effort, because many of the team members weren’t on the project from the start! Pausing multiple times along the project lifecycle allows the team to apply the learnings as the project is progressing, rather than waiting until the end where the opportunity to improve has passed.
- Establish a centralized repository to track advancement of the key learnings and share results. Identify at least one resource (typically it is the project manager) to be responsible for tracking the learnings. The mistake I made is that I provided a summary report to the team after the retrospective, but I didn’t have a process defined to ensure a commitment was made to use the learning and no tracking process other than follow up notes within my Outlook calendar and copies of the collateral on my local hard drive. We have fixed this problem by developing a customized Share Point site with all the data and learnings consolidated into one place. We implemented a sophisticated search engine to easily find learnings and designed the site with no training required to use the database. The goal is to ensure the Recommenders (experts on the issue) are connected via the repository with Receivers (those who can use the learning) who Commit to doing something different and the Results are tracked. We use the term RRCR to describe this process.
- Monitor the progress on the action items via an automated workflow. Other priorities didn’t allow me to follow up with all the teams in a timely fashion. After three to six months I would informally ask the project managers how things were going, but I didn’t have the authority or responsibility to track the individual team efforts. We have set up an automated workflow within SharePoint to send reminders to the teams periodically so we can keep adding new data to the learnings in a timely manner.
- Communicate the status and document results from the action plans. We have a lot of interesting data and no process to effectively share it. We set up a process for sharing BKM’s with project managers via our institutionalized Corporate Lifecycle framework where detailed entry and exit criteria designed to remind teams to add learnings into the database and to commit to use learnings to improve their current projects. We also have established a quarterly briefing to senior management on the continued progress and share results. Communicating results to management on the status of improvement actions is essential to continuous process improvement. Remember the old adage: “What gets measured gets improved.”
My job provides me the opportunity to collaborate with a wide variety of project managers all over the world to help them tip over rocks to expose “squiggly things” in a way that is fun, effective, and contributes directly to the continuous process improvement efforts at Intel. Luckily, the retrospective framework allows me the platform to discuss all the “scary” things in a way that directly improves the effectiveness of the team. This in turn, positively impacts the success of Intel Corporation. I urge you to go forth and begin using retrospectives in your organization today. The squiggly things are worth uncovering and dealing with.
Collins, James C. (2001). Good to great: Why some companies make the leap…And others don’t. New York: HarperCollins Publishers.
Derby, Esther, & Larsen, Diana. (2006). Agile retrospectives: Making good teams great. Pragmatic Bookshelf.
Kerth, Norman L. (2001). Project retrospectives: A handbook for team reviews. Dorset House Publishing Company, Incorporated.
Sample e-mail to introduce the pre-work survey
In preparation or our Evolution 1.6 project retrospective, I need your assistance with completing a short survey (actually it is pre-work for our session) to give us a baseline for our discussions. We are fortunate to have a neutral, trained facilitator, Debra Lavell, who will be leading us through the process. The survey invite and link will come directly from Debra and all responses will be confidential and anonymous. Please be open, honest, and constructively critical so we can focus on what went right and how to capitalize on opportunities moving forward.
Our first session is scheduled in two weeks, on Monday, August 3rd from 3:30-5:30pm PST. The second session will be held from 7:30am-9:30am on Wednesday, August 12th. I look forward to productive sessions. Thank you for your time and dedication to the process.
Project Leader’s name & title goes here
Action Planning Template
- Problem Statement: Create a clear, concise problem statement and identify the impact to the project. For example: Weak problem statement: Team did not define requirements very well
Better problem statement: Lack of x, resulted in y. For example: Lack of timely feedback from customers (usually received late in program planning) results in requirements changes after they were approved and put under change control.
How did this effect work on this project? Cost, schedule, quality, productivity, morale?
- Recommended solution(s) Advantages/Disadvantages of implementing the recommended solution.
- Discuss the impact of solving this problem. Will solving this problem save time or money? Increase effectiveness, productivity, or morale?
- Uncover the obstacles. Who or what could get in the way of implementing the proposed solution?
- Identify who needs to support this recommendation
- Establish next steps with clear owner and set due dates for moving the solution forward
Sample agenda for first session to identify key learnings
|• Duration||• What||• Expected Outcome|
|• 10 min||Introductions and set objectives||Enable an effective meeting|
|Align on purpose of the retrospective|
|• 10 min||Quick program overview||Scope, value proposition and timeline of the project|
|• 60 min||Review survey data to identify key messages||Synthesize into key learnings|
|• 30 min||Prioritize top learnings||Identify recommended focus areas and common themes|
|• 10 min||Wrap up and set next steps||Discuss how the team will create action plans in the next meeting|
Sample agenda for second session focused on action planning
|• Duration||• What||• Expected Outcome|
|• 20 min||Review key messages from session one||Enable an effective meeting|
|Align on purpose of this second meeting|
|• 90 min||Write action plans||Fully documented action plans the team can be proud of and execute on|
|• 10 min||Wrap up and set next steps||Share how the outcomes from today will be implemented and communicated|
© 2010, Debra Lavell
Originally published as a part of 2010 PMI Global Congress Proceedings – Washington D.C.