Dynamic application of lessons learned from a large business transformation
Chet Chetzron, Partner, CSC
Our presentation will focus on how to apply lessons learned gathered from some of the largest transformation efforts in the federal marketplace. The unique challenges faced in supply chain transformation enabled by Enterprise Resource Planning (ERP) Systems require an effective lessons learned approach. There are several modernization efforts underway, aimed at integrating best commercial practices like ERP into the federal government supply chains that require us to look beyond traditional under applied lessons learned approaches and leverage real time learning. We will discuss the importance of maintaining a dynamic lessons approach for information technology efforts to meet the challenging mission of the Federal Government well into the future. Since the publication of the 25 Point Implementation Plan to Reform Federal Information Technology Management and the legislation of the Government Performance Reform Act of 2010 (GPRA), the federal government will undoubtedly undertake projects to implement some of the most desperately needed changes and transformations. As budgets grow tighter and service expectations rise, these projects will be more closely monitored for budget, schedule, and delivery of promised functionality. Keeping those projects on the right track to deliver value will require leveraging previous lessons learned and an effective way to integrate them into current and future opportunities.
Why Lessons Learned Are Important
The Department of Defense (DoD) has made a significant investment in ERP solutions over the past decade. Spending has totaled US$9 billion on 14-15 ERP solutions, and until recently, less than 10% of total obligation authority was being managed by these systems (Fisher, 2010). The magnitude of time, resources, and investment of capital require that the decision to undertake an enterprise transformation using an ERP system or other tools is made with careful consideration and forethought. For many organizations, the payback outpaces the risks. As we look back on many ERP projects and their varied outcomes, we tend to forget that they begin with the same set of application compact disks (CDs). Why are there significant differences in the outcomes of these projects? Some enterprises are wildly successful in achieving the promised benefits, and others are noted by failure. While scope and complexity have a marked impact on transformations, the manner in which programs are able to learn and adjust to their unique set of challenges will increase their opportunity for success. Government managers must learn from what is working in the private sector and apply these best practices to delivering services better, faster, and at lower cost (Carney, 26 April 2011, Sec.1).
The contribution and importance of lessons learned are recognized throughout the DoD. The joint services all support input to the DoD Joint Lessons Learned Information System (JLLIS). JLLIS enables continuous process improvement by gathering observations and recommendations and making those data available across a wide range of agencies such as Defense Logistics Agency (DLA), Defense Information Systems Agency (DISA), Defense Threat Reduction Agency (DTRA), Department of State (DOS), Army, Navy, Air Force, and Marines. The services must standardize their inputs so they can be used in JLLIS. The Air Force Center for Knowledge Sharing Lessons Learned is responsible for managing lessons learned for the Air Force to enhance the combat effectiveness of all Air Force units. The Center for Army Lessons Learned facilitates rapid adaptation initiatives and conducts focused knowledge sharing and transfer that informs the Army of efficiencies. The Navy Lessons Learned Program (NLLP) provides the Navy with a structured process to capture and analyze lessons submitted by operational forces in order to improve fleet readiness and to enhance the Navy’s ability to accomplish assigned missions. These are mostly web enabled and are closely managed.
In accordance with the GPRA Modernization Act of 2010 (31 U.S.C. 1115 et seq.), each agency’s Chief Operating Officer (COO) shall be designated as the Senior Accountable Official responsible for leading performance and management reform efforts and for reducing wasteful or ineffective programs, policies, and procedures (Carney, 13 June 2011Sec. 2c). Many organizations hire integrators to perform the consulting and systems integration work to leverage a consolidated version of best practices gained and learned from years of projects and client engagements; however, depending on the scope of the project or program, this may not be sufficient. Large programs that include design, development, and deployment phases must have a dynamic lessons learned program established so that unique or specific lessons learned from this program will be documented and implemented very soon after they are identified to avoid repetition of mistakes in future phases of the program.
Dynamics of Lessons Learned
Lessons learned constitute organizational learning captured from performing work on a project or across programs or DoD agencies when communication and feedback mechanisms are in place. Lessons learned may be positive or negative. The avoidance of negative lessons learned and further leverage of positive lessons learned benefit a project, program, or organization through increased productivity and reduced cost. They must be properly identified, articulated, and then effectively integrated back into the process.
Application of lessons learned is a tool for improving ERP deployment readiness. Enhancing probability of success requires three key areas of people, process, and technology all be nurtured during the transformation. Many transformations fail to pay attention to the human factors, but motivation for learning during the project or programs is very complex, and it requires a strong adherence to project management disciplines. For example, many organizations that chose to implement ERP ahead of Year 2000 (Y2K) had to contend with competing priorities. With a fixed Y2K deadline looming, many organizations were faced with a choice between schedule and process excellence, and unfortunately, schedule prevailed. Racing to meet a Y2K deadline turned out to be a costly decision for many firms that did not take the time needed to understand organizational goals and business process.
Reasons for Success
Many reasons for ERP transformation success (or failure) have been identified and categorized through a lessons learned processs. Project Management Institute (PMI) regards lessons learned as organizational process assets. Since best practices and lessons learned are foundational to project management maturity, most successful transformations will include aspects of knowledge transfer in their planning. With so many of these lessons already known and identified, programs have the potential for initiating with a planned improvement baseline. Many successful programs are also recognizing “discovered” improvements identified after an activity or event has occurred by initiating an action to gather the “discovered” information. When professionals perform collection and identification of lessons learned, they will aggregate and rationalize responses into meaningful themes that provide transformation executives with focus areas. When programs fail to meet cost, scope, or schedule targets, we can attribute a disregard to one or more of the key transformation areas. Here are several important categories to note derived from lessons learned:
Executive Leadership Alignment and Accountability for Success
Well Documented Vision and Business Case for Action
Careful Selection of Software and Vendor
Experienced Program Management and Control
“Best and Brightest” Resources Assigned to the Program
Formal Data Clean-Up Program
Ongoing ERP Education Program
Comprehensive ERP Training Program
Commitment to Process and Organizational Change
IT Tools for managing changes to Scope, Schedule, and Cost
Since many of the lessons learned from various programs are acknowledged in publications, the main differentiator becomes how “discovered improvements” are identified, applied, and made reusable. There is always a tendency to proceed along the path of least resistance to continue the same process with modification as opposed to making a transformational change. That is why management should support ongoing transformational change, as opposed to cosmetic change, as an expected outcome of performing lessons learned. Dynamic application of the lessons learned process is what we will address in the following pages.
Planning Process Group
The Program/Project Management Plan identifies the Functional Process and Risk Management team as the owners of lessons learned (Exhibit 1). This includes, but is not limited to, developing a robust Lessons Learned Process. As we start with lessons learned from previous projects, our transformation process is adjusted over time so that we take advantage of practices that have proven effective and do not repeat costly mistakes. Organizations can better leverage lessons learned in guiding implementation by ensuring activities for capturing and disseminating lessons learned are a part of the project plan. It is vital that transition teams contribute to the improvement of the transition approach by recording lessons as they are learned. The lessons learned process should be program-wide.
Exhibit 1 - Lessons Learned Process Owners
The Planning Process Group is of critical importance for lessons learned, since time constraints, delivery scope, and budgets are addressed here. Most post-performance collection analyses may provide meager results because they were not captured early on; therefore, an opportunity was lost to have them implemented during the project, perhaps in future phases of the project. Lessons learned activities, process, training and templates, are identified by the Planning Process Group. The lessons learned process should be a continuous one that is formally addressed on a regular basis during the program/project. In this way, learned lessons are documented when they are fresh on team members’ minds, shortly after they are realized.
The Process Improvement Plan (PIP) defines the activities and priorities for process improvements documented in the Process Baseline to support program goals. Steps for guiding process improvement and lessons learned are contained in the Program/Project Management Plan. Processes are standard, repeatable ways to perform work and contribute to achieving organizational goals and objectives.
At CSC, we use our Catalyst methodology (Catalyst) as a foundation for the Process Management Approach. Specifically, process management ensures that an organization’s processes incorporate continuous improvement through feedback and lessons learned from process executions. Lessons learned are one input for the process management approach.
Lessons learned are a source of potential process improvements (Exhibit 2) along with contract changes, measurement analysis results, benchmarking results, and feedback from process users.
Exhibit 2 - Lessons Learned Output
Process improvements may be also identified in the course of evaluated processes. Post-performance analysis is supported by the evaluation of Process Improvements. As the program moves along the life cycle, the goals and needs of the program will change and as such, the lessons learned will address those needed changes (Exhibit 3).
Exhibit 3 - Lessons Learned Drives Process Improvement
Setting the Stage for Finding Lessons Learned
Executing Process Group
When the need to develop lessons learned is triggered, the manager or director who owns the related work area sponsors lessons learned meetings. The lessons learned sponsor takes an action to appoint a facilitator. Preliminary interviews are conducted with program participants and key stakeholders. The sponsor should define topics and objectives for the upcoming lessons learned meeting and assure that the facilitator understands the topics and is acquainted with the lessons learned meeting participants. The facilitator can then take a more active role in determining the format to collect lessons learned, such as a survey or a meeting. The facilitator will identify roles, such as recorder and subject matter experts, and ensure the proper materials are available for the meeting, which may include overview documentation and a lessons learned preparation template. Awareness training is performed to increase organizational understanding.
Dynamic Lessons Learned Improvement Process
Monitoring and Controlling Process Group
As project activities are monitored and controlled, the process for collecting lessons learned data and information is conducted. Triggers and identification of opportunities to capture lessons learned from stakeholders and project team members are leveraged in future phases or post-deployment activities. Appropriate data collection and feedback mechanisms should be in place to support a continuing focus on lessons learned.
Capturing Lessons Learned
Lessons learned are captured throughout the project life cycle. Establishment of that culture and mindset throughout the project is critical for a dynamic process. Although lessons learned have a role in all five of the PMI Process Groups, we have focused on Planning, Executing, Monitoring & Controlling, and Closing. Examples of when lessons learned capture takes place is are listed below:
For project work, this process is conducted, at a minimum, at the end of each project sub-phase, phase, and/or milestone.
For program-level and other support work not related directly to the project life cycle, this process will be conducted by each organization in conjunction with the Rolling Wave Planning cycle—occurring at regular, defined intervals in the planning cycle on an ongoing and iterative process.
For work conducted by a tiger team for a specialized need identified by a manager, as a qualified lessons learned opportunity on an ad hoc basis.
For special events or specific purposes not covered above, this process may be conducted at any time at the discretion of a manager or director.
There are several channels for identifying the lessons learned. The most prevalent of these is the use of a lessons learned meeting. Initial capture of lessons learned may be done virtually using the Web or email (Exhibit 4). The data elements collected are as follows:
Lessons learned ID
- Identifying Team
- Recommended Responsible Team
- Date Identified/Meeting date
- Need By Date
- Governance Tool (Risk, Issue, Action Item) or Disposition
- Reviewed by Lessons Learned Team
- LL Benefit (Maximum, Moderate, Minimum)
Exhibit 4 - Lessons Learned Capture Tool/Template Example
Conduct Lessons Learned Meeting
This process step entails conducting a meeting according to the published agenda for the purpose of eliciting lessons learned pertaining to the subject topics. During this meeting:
- The facilitator solicits participants for specialized roles, such as a timekeeper or recorder who assists with conducting the meeting.
- The facilitator ensures that accurate attendance is recorded (including any remote participants), provides participants with any necessary materials, and reviews the meeting objectives, agenda, methods, and roles.
- The facilitator and sponsor decide if participation will be shared and open to all stakeholders or limited to internal participation only. However, all lessons learned can be shared with the agreement from the associated manager responsible for a work stream.
- The facilitator encourages observations from the participants regarding pertinent lessons that should be documented and, if necessary, spurs discussion on validity of the suggested lessons.
- The facilitator should pay special attention to ensure that both positive and negative lessons learned are brought forward in discussion.
- The facilitator assists participants in honing initial suggestions into a set of lessons that are significant enough to capture for the benefit of others in the future. Formulation of specific lessons from observations may be performed by the facilitator (in conjunction with the sponsor, if necessary) after the formal meeting if time does not permit doing so during the meeting.
Those lessons learned must not be:
- Information that does not convey new knowledge
- Information that cannot be put to practical use in the future
- Restatements of an existing policy or process
- Observations (e.g., “Project schedules were always inaccurate” is an observation, not a lesson learned)
- Imperative statements (Lessons learned should include explanation that fosters understanding of why the lesson was learned, e.g., “Always resource-level project schedules” is an imperative, not a lesson)
- Trivial information others are likely to know
- Judging performance or results
- Evaluations of the performance of people
Face-to-face meetings conducted in conference rooms are not required. Virtual meetings or meetings conducted using various methods of collaboration may be used if expedient and practical.
Here are the items to be included in the lessons learned summary report:
- Lessons Learned Subject/Title
- Date Conducted
- Sponsoring Organization
- Lessons Learned Sponsor
- Triggering Event
- Lessons Learned Type
- Can this Lessons Learned be shared with the client/other stakeholders?
- Name of Approving Manager
- Attendees (Internal/External)
- List of Participants
- Improvement/Resolution Assignment
Once the initial list of lessons learned is identified and reviewed, each lesson learned must be updated and categorized into the appropriate project work stream, and then the major lessons learned are documented in the lessons learned database (DB). There are a few key tags that need to be added to guide appropriate assignment in the program for tracking and resolution. Not all lessons learned will be actionable. Program level process lessons learned will have broad-based consequences to the program if they are not mitigated appropriately or if improvements are not implemented. Any lessons learned that will not benefit the program as a whole, but will specifically benefit one of the project teams are tracked and managed at the team level. These are the lessons that will move forward to be used for the benefit of others. A member from a department project/program organization is assigned as the owner.
The Lessons Learned Process feeds the Action Item Process and the Improve Process by identifying opportunities for improvements. Post-performance analysis is supported by the Improve Process Description Process and the Lessons Learned Process. Lessons learned identified as program level lessons flow directly in to the Improve Process stage as documented Suggested Process Improvement (SPIs) or the Action Item Process (Exhibit 5). Program Level Lessons Learned are categorized as either a formal SPI or an Action Item. Action Items are managed by the Governance Team and SPIs are managed by the Process Management Team. Action Items may be resolved at the owning team level, or the team may request escalation to the Issues or Risk Management governance area for subsequent tracking.
Exhibit 5 - Lessons Learned in Relation to other Processes
Risk Management and Lessons Learned
Lessons learned can serve as one of the inputs to the risk identification process. These inputs are evaluated for use in risk analysis needed to support major program decisions during the program life cycle. A quick review of lessons learned from earlier assessments, combined with abbreviated versions of these suggested steps, will allow the organization to build on previous learning.
There are three main components related to the lessons learned capture and dissemination:
- Lessons Learned Database
- Action Item Tool
- Suggested Process Improvements Tool
Exhibit 6 - Lessons Learned Capture Database
The facilitator and key stakeholders must establish the initial list as part of the lessons learned DB. Each lesson learned must be added into the appropriate repository (Exhibit 6). Our process includes an internal and external use database for the purpose of encouraging unguarded feedback. If participants omit comments or provide diluted comments that are constructed to avoid offending others, we will not have a useful product. For the process to be effective, it is important to obtain candid feedback.
The lessons learned are reassigned to the appropriate tool for ongoing tracking. The sponsor and participants should review these items on a frequent schedule to ensure the dynamic inclusion of important lessons learned into the current or future work streams.
This process step entails post-meeting activities for documenting and disseminating the results of the Lessons Learned process.
- The facilitator completes the preliminary Lessons Learned Report, using the appropriate Conduct Lessons Learned Report Template (see Exhibit 4). The Lessons Learned Report is sent to the participants and other stakeholders for review and comment.
- After responding to any participant comments, the facilitator sends the Lessons Learned Report to the sponsor for review and comment.
Project or Phase Closing Activities
Closing Process Group
Often, lessons learned will be part of the check list for project closeout. Lessons learned closing activities are also identified as part of the end of a phase or sub phase. Having lessons learned stored in an accessible location for the future shows a commitment to transformation and continuous process improvement.
- After responding to sponsor comments, the facilitator stores the Lessons Learned Report, along with any supporting documentation, in a new folder in the lesson learned DB.
- The facilitator emails participants and the sponsor with the link to the stored Lessons Learned Report. The facilitator also sends email notification to appropriate non-participant stakeholders that lessons learned concerning the subject topic are available.
A close-out session with team members, a presentation to program management, and posting to the program portal are all important elements of the process group. Close-out sessions can vary in approach and content, depending on the audience. Reviewing the Lessons Learned Report and identifying these critical success factors with the program executive team and other members of the program management team is highly encouraged. The report provided to the program executive team should contain the following information:
- Executive Summary. The Executive Summary is aimed at a few key leaders who may not read any other part of the document. Since the Executive Summary is written for a specific audience, the orientation and level of detail will vary from client to client. This section summarizes the lessons learned and recommendations addressing root causes, sustaining critical success factors, and continuing the lessons learned process.
- Stakeholder Feedback. This section summarizes the results of any stakeholder program review assessments (interviews, focus groups, surveys, and so forth). Describe the stakeholders who participated in the assessments and highlight key findings.
- Lessons Learned Themes. This section includes the analysis of stakeholder feedback (typically, the results of a lessons learned workshop). Identify recurring themes and root causes of issues and problems. Include recommended actions that would address root causes and prevent these problems from recurring on future business change programs.
- Critical Success Factors. This section identifies the critical success factors suggested by the assessment data. Include recommended actions that would increase the likelihood that these critical success factors would be in place for future business-change programs.
- Recommendations. This section includes recommendations for reviewing the lessons learned with key stakeholders, documenting the lessons learned in the organization’s knowledge management system, and designing a process for documenting and reviewing post-implementation lessons learned.
We are constantly striving to improve our programs and processes. In order to maintain the effectiveness of our approach, we perform reviews to obtain sponsor and facilitator input. Performing lessons learned on Lessons Learned (LL2) allows us to capture unique insight that we can later share with our professionals and will ultimately benefit our clients or stakeholders. Like many guidelines and process documents, the lessons learned process that looks great on paper does not always translate to meaningful results. In order to insure continued improvement, we identify key themes to share with the program executive team. These LL2 insights are provided to sponsors and facilitators for consideration as future lessons learned work effort is planned and managed. Here are improvement themes that should be taken into account:
Time and money. These tend to be the two biggest factors in mounting an effective lessons learned program. The administrative effort associated with lessons learned should not be underestimated. Once a program is funded, it is important to ensure that the positive and negative lessons learned are being communicated. The type of contract, whether firm fixed price or time and materials, for example, can influence how lessons learned activities are resourced in the project plan. Fixed price contracts must have this effort and resource requirement factored into program budgets. Time and materials contracts should at least ensure that the effort is identified and agreed on during the planning process group.
Do not punish the messenger. Since negative lessons learned tend to be a list of failures, we are sometimes selective with what is passed on to our enterprise. Executive management should understand these sensitivities regarding negative lessons learned and guard against using “self reported” lessons as a hammer. For the process to be effective, it is important to have this candid feedback even at the risk of offending others along the way.
Consistency in application of the process. Lessons learned process teams should be dedicated to ensuring consistent interpretation of lessons learned and administrative support. Avoiding a matrixed approach to completing lessons learned will minimize the interpretation factor used when the knowledge ware and guidelines are interpreted. Making lessons learned an executive priority will allow users and professionals to align competing priorities. Key players that were involved with the activities can focus on the improvement ideas while the lessons learned process team can focus on the administrative activities related to identifying and rationalizing lessons learned.
Skin in the game. Required inputs for lessons learned need to be well defined and enforced by lessons learned professionals. Insist that delivered lessons learned are fully developed and well thought out. Quality is more important than quantity. When users are not willing to make the time investment to produce well thought out lessons learned, it is doubtful they would survive the rationalization process, resulting in unnecessary cycles.
Avoid the big bang at the end. Push for collection throughout the project life cycle at key, defined points. This concept is well regarded but can provide some challenges, depending on the level of program/project executive support. Improvement ideas tend to be more scattered and less focused as time elapses. The risks of losing resources and shifting priorities increase at the end of a phase or sub phase.
As we strive to make the lessons learned process dynamic and integrated, the key to success resides in establishing a framework that empowers subject matter experts and lessons learned professionals. Having the proper organizational behavior in place is vital for meaningful lessons learned to be documented and well understood. Supporting process activities are conducted in a disciplined manner throughout the project life cycle, with the use of tools, templates, and methodologies that support a continuous process. Tools and discipline are the pillars for sustaining transformational change. With leadership support firmly in place the information shared through and across entities will enable valuable process enhancements. Mandates for efficiency improvements facing federal agencies will necessitate that the resulting transformation projects require lessons learned an essential element of future programs.
“Those who cannot remember the past are condemned to repeat it.” (Santayana, 2005, p. 92).
Aha, D.W. (2001, November). Local lessons process: A radical proposal for sharing lessons within the DoD. Retrieved from www.aic.nrl.navy.mil/aha/ida
Air Force. (2010, September). Air Force Instruction AFI90-1601 Air Force Lessons learned program. Retrieved from www.e-Publishing.af.mil
Carney, J. (2011, April 27). Office of the Press Secretary, The White House, Executive Order-Streamlining Service Delivery and Improving Customer Service. Retrieved from http://www.whitehouse.gov/the-press-office/2011/04/27/executive-order-streamlining-service-delivery-and-improving-customer-ser
Carney, J. (2011, June 13). Office of the Press Secretary, The White House, Executive Order-Delivering an Efficient, and Effective, and Accountable Government. Retrieved from http://www.whitehouse.gov/the-press-office/2011/06/13/executive-order-delivering-efficient-effective-and-accountable-government
Fisher, D. (2010, April 2). ERP Lessons Learned Panel. Defense Logistics Modernization sponsored by the Center for Strategic and International Studies. Retrieved from http://csis.org/multimedia/audio-defense-logistics-modernization-panel-1-logistics-erp-%E2%80%93-lessons-learned
Free, D.L. (2000) Air Force Center for Knowledge Sharing Lessons Learned, AAAI Technical Report WS-00-03. Retrieved from http://www.aaai.org/Papers/Workshops/2000/WS-00-03/WS00-03-005.pdf
Ganly, D. (2008, August 29). Address Five Key Factors for Successful ERP Gartner research ID Number: G00160011 retrieved from www.gartner.com
Gattiker, T., & Goodhue D. (2005, September). What happens after ERP implementation: Understanding the impact of interdependence and differentiation on plan – level outcomes. MIS Quarterly. 29(3), 559–585.
GPRA Government Performance and Results Act. P.L. 111-352 H.R. 2142. Retrieved from http://thomas.loc.gov/cgi-bin/query/z?c111:H.R.2142:
Herbert, L. (2008, July 23). Successfully managing ERP implementation providers Forrester research for sourcing & vendor management professionals. Retrieved from forrester.com
Kundra, V. (December 9, 2010). 25 Point implementation plan to reform federal information technology management. Retrieved from http://www.cio.gov/
Project Management Institute. (2008). A guide to the project management body of knowledge (PMBOK® Guide) (4th Ed). Newtown Square, PA: Author.
Santayana, G. (2005). The life of reason [Electronic Version] Retrieved from www.gutenberg.org
Williams, T (2007). Post-project reviews to gain effective lessons Iearned PMI. Retrieved from PMI.org
©2011, George Borza & Chet Chetzron
Originally Published as Part of 2011 PMI Global Congress Proceedings – Dallas, TX