The Software Engineering Institute's Capability Maturity Model (SEI, 1998) is synonymous with software engineering quality in many organizations. It is based on the assumption that organization software engineering process maturity can be assessed against a standard. The CMM is that standard. The goals of the CMM are improved software quality, reduced software development cost, and decreased time to delivery of engineered software products. The CMM defines software engineering process maturity at five levels.
1. Level 1 is the initial state. Level 1 organizations are undisciplined and often chaotic. Software engineering success, when it occurs, is the result of heroic efforts on the part of individuals or workgroups.
2. Level 2 is a repeatable state. Level 2 organizations have policies, plans, procedures, training, assessments and metrics in place to assure that software engineering successes can successfully implement similar projects. Six areas of discipline or Key Process Areas (KPA) are implemented: Requirements Management, Software Project Planning, Software Project Oversight and Tracking, Software Subcontract Management, Software Quality Assurance, and Software Configuration Management.
3. Level 3 is a defined state. Level 3 organizations are characterized as having standard processes for developing and maintaining projects documented and used across the organization. Projects tailor their software development processes (PDSP) to the organization standard software process (OSSP). Seven additional areas of discipline (KPAs) are added to the six implemented at Level 2: Organization Process Focus, Organization Process Definition, Training Program, Integration Software Management, Software Product Engineering, Intergroup Coordination, and Peer Reviews.
4. Level 4 is a managed state. Level 4 organizations assess software engineering performance, and product and process quality against performance standards to achieve measurable goals. Level 4 adds discipline in two areas: Quantitative Process Management and Software Quality Management.
5. Level 5 is an optimizing state. Level 5 organizations are focused on continuous improvement of software engineering process, change management, and defect prevention. Three KPAs are added to the discipline previously implemented. These include: Defect Prevention, Technology Change Management, and Process Change Management.
An SEI CMM implementation team frequently encounters an obstacle that can compromise or even doom the success of the program: The time required to bring an organization to a higher level of maturity exceeds the patience and budget of the corporate leadership team.
Project Profile
This case study describes a rules-based Rapid CMM Attainment Technique (R-CAT) used to reduce the time required to implement a Level 3 process infrastructure and compares this method against two other attainment approaches commonly used in SEI CMM implementations.
The Company
The company discussed in this case study is a major provider of engineered software projects to the telecommunications industry. Corporate leadership initiated an SEI CMM implementation program designed to deliver CMM Maturity Level 3 discipline over a two-year period. The goal was a 15% reduction in maintenance cost by the elimination of maintenance related to production stopping defects.
The company was organized into several self-managed development centers with three horizontal support centers providing services to all centers. For the purpose of CMM, each center would be considered an organization.
The SEI CMM attainment mandate applied to development and support centers equally. Centers were charged with the implementation of a Maturity Level 2 infrastructure within one year of the program kick-off. Level 3 infrastructure would follow in year 2 of the program. Because a CMM Based Appraisal of Internal Process Improvement (CBA-IPI) requires that an organization operate at a maturity level for at least six months after instantiation of the infrastructure, the infrastructure implementation would be considered a separate milestone from the full CBA-IPI organization assessment.
To increase the speed of attainment, leadership encouraged centers to choose consulting firms to partner in the implementation effort. Because centers selected their favorite consultants multiple transformation approaches were introduced across the company.
Exhibit 1. R-CAT Framework
The Organization
The center in this case study was one of the horizontal support centers. It consisted of 350 people organized into 17 project teams divided among five directors. The R-CAT initiative began after another consulting company, having tried and failed in the transformation effort, was asked to leave the organization.
Being “last out of the gate” frustrated the center's management team. Skeptical of success and facing an overwhelming bias against the approach by the corporate coaching team, the center reluctantly proceeded. They had no other alternative. With little reason to believe that the center could achieve corporate CMM attainment objectives in the six months remaining in the initiative, the center selected another consulting team to partner in the effort. As the CMM attainment program ground to a halt halfway through year one of the initiative, organization directors were troubled by several concerns:
• The organization had lost half the year in a failed attempt; the directors believe that the center could not complete its attainment mandate in the time remaining.
• A significant portion of funding committed to the change initiative had been spent.
• The consulting company retained to provide leadership coaching at the corporate level had advised the corporate leadership team that the CMM attainment effort would fail in the center for lack of time.
• One director and pockets of hostility to the CMM attainment initiative were scattered throughout the project teams.
• Having lost faith in one consulting company, the directors were naturally reluctant to cooperate with yet another outside consultant.
• The program had lost critical momentum with no guarantee that forward progress could be regained.
• The organization could be characterized as highly change resistant.
The consulting team selected had less reason to be pessimistic. Prior engagements demonstrated that when a Rapid CMM Attainment Technique (R-CAT) is applied under the guidance of seasoned facilitators, organizations can not only implement maturity Level 2 infrastructure within six months, but can implement Level 3 maturity infrastructure within that time frame as well. Once the infrastructure is constructed, it remains for the organization to continue applying the Level 3 policies, tools, processes, procedures and disciplines for six months to assure a Level 3 assessment on their CBA-IPI.
The Approach
As Exhibit 1 shows, R-CAT is a rules-based transformation methodology structured around an interlocking combination of teams, facilitation, process engineering, program management and governance.
Teams: The center divided itself neatly into teams based on director. Each team consisted of eight to 12 technical leads responsible for one or more projects. Each project team member committed to spend four to six hours per week on the attainment effort and assured that the project staff under them would devote one hour per week per person to the attainment effort.
Best Practices: One director managed a cross-functional team composed of eight subject matter experts from all the other teams. This team took responsibility for documenting in procedural format software engineering best practices currently in use in the organization. Another project team had processes that were largely maturity Level 2 compliant. It had taken this team of 20 software engineers 18 months to document their processes with the help of a process engineer. This team agreed to donate their existing process specifications and templates to the organization as a process baseline and to pilot processes, practices and templates required for the Level 3 organization infrastructure.
Facilitation: Facilitators from the consulting firm were served as coaches to two teams each. Facilitators had previous experience delivering change using the R-CAT methodology. A CMM advisor (a consultant from the leadership coaching group) provided expertise on the Capability Maturity Model.
Process Engineering: Process engineers rounded out the consulting team. Two PEs were assigned to each team.
Governance: Another director assumed dual responsibility. He provided administrative coordination of the organization attainment effort in addition to his role in a process team. A governance board managed escalations and kept the initiative on course. Directors, facilitators and the CMM advisor sat on this board. The board met once a week for two hours.
Program Management: A two member Program Management Office designed the Process Component Library and the web site. Once The PMO collected and communicated measurements on process and quality.
Assessments: At the start of the initiative and periodically throughout the program, process engineers under the leadership of the CMM advisor, assessed projects against the CMM and documented organization progress toward CMM attainment objectives.
Iterative Cycles: The six months remaining in the attainment period was divided into three cycles separated by two weeks in which no team activity was allowed to occur. These rest periods were deliberately designed into the program to allow project management staff to rest from the intensity of the program and to allow the changes generated in each cycle to be assimilated into the organization.
Training: As part of the kick-off every member of the organization (350 people) and all members of the change team received eight hours of training in organization behavior theory. During training staff assessed the organization culture and explored organization rules that could jeopardize CMM attainment. They learned how organization rules managed the behavior of individuals and groups. They discovered how individuals could use the new CMM framework to ensure personal success, and how to tailor the corporate ruleset to assure the implementation of a Level 3 process infrastructure within six months. They learned that their organization could leave the gate last and still complete the program in the allotted time.
Implementation
Cycle 1: “As-Is” Documentation
During the first program cycle, teams documented their “as is” processes. “As-is” processes are present method of operation—the processes currently in use to deliver software. In a Level 1 organization these processes are rarely systematically documented and maintained. Although the governance board expected that teams would use the baseline processes contributed by the Pilot team as basis against which to document current processes, teams did better documenting their processes “from scratch” on a white board.
The first step in documenting “as-is” processes is to divide up the process engineering work into logical increments. For this project, the teams elected to the process documentation effort into the six increments corresponding with the six phases of their software engineering life-cycle. By the end of Cycle 1, teams added two additional increments:
1. Management processes that span multiple software engineering life cycles (e.g., funding, staff procurement, and training).
2. Infrastructure processes that represent technical activities completed frequently throughout the life cycle (e.g., configuration management and peer-reviews).
Under the guidance of facilitators, teams documented one process increment per week. Facilitators encouraged their teams to document processes that were actually in use in the organization. In each session, facilitators helped their teams examine organization rules, technology constraints and management structure that supported their processes.
This exercise was particularly beneficial when “as-is” processes did not seem to make sense because they failed to produce desirable technical, quality and performance results. It provided insight into cultural issues that could undermine CMM attainment objectives. It also helped coalesce the team into a high-performing unit. By the end of Cycle 1, most teams worked well together, and the team members themselves discouraged changes in team composition.
The objectives of Cycle 1 were clear: documentation of “as-is” process only. However, as team leads documented “as-is” processes, they discovered flaws that they wanted to stop and correct. In consequence, the goal of Cycle 1 was often met with resistance from the process teams. This put facilitators under constant pressure to keep their teams on track.
Between process meetings, Process Engineers formalized team processes. When the PEs released a formalized process team members socialized the processes with their staff. Each staff member was encouraged to review, comment on, and test the “as-is” processes against their actual work activities.
On a weekly basis, the PMO documented and reported to the governance board the number of processes documented. In addition, they recorded actual time spent by the organization on attainment activities. The target was and 4–6% total work effort distributed across the organization staff and 100% staff participation in weekly attainment activities.
Midway through Cycle 1, the transformation team made a sobering discovery. The technical leads of one project team had a strong bias toward Organization Development (OD). This team disagreed with R-CAT and believed that an OD intervention approach (with its focus on change management) would deliver results more quickly for their part of the organization. They staged a revolution, dismissed their facilitator and process engineers and restructured their attainment effort along more classical OD lines. Their focus centered around leadership skills, training, psychological profiling and the management of human factors issues, rather than process engineering and modification of organization rules. In a move that would later be regretted, the Governance Board, under pressure from the director, took a hands-off position, provided a new OD facilitator to the team and brought in two new process engineers. With new consultants on-board, the maverick team closed lines of communication between their project and the rest of the organization.
By the end of Cycle 1, three out of four teams had documented “as-is” processes that closely resembled processes that worked within the constraints of the organization culture. Although the teams had reverted to a “blank page” approach, their “as-is” processes looked remarkably like those of the Pilot team. The best practice group ended Cycle 1 with a defined scope of work and a Best Practice Suite for Peer Review. The Level 3 pilot group was well under way, enriching their existing processes with Level 3 competencies. These policies, procedures, practices and templates would, by the end of Cycle 2, mature into the organization Level 3 infrastructure. The OD team extended Cycle 1 for four weeks, putting their CMM attainment goals at risk. Without insight into the activities of the OD team, the Governance Board had little understanding into their progress as Cycle 1 came to an end.
Between cycles, while the project teams rested, took vacation, and focused on their real work of software development, intense activity occurred behind the scenes:
• Process engineers wrote process specifications to augment “as-is” process flow charts.
• The CMM advisor assessed “as-is” processes against CMM Level 2 common features.
• Facilitators prepared a gap analysis by life-cycle phase for each team, identifying process weakness to be corrected during Cycle 2.
• Facilitators also created templates for a Project Plan, Software Development Plan, Project Training Plan, Software Quality Assurance Plan, and Software Configuration Management Plan.
• The Best Practices group released a detailed procedure for Peer Reviews including quality checklists and review templates.
• The PMO updated the metrics database and stored baselined “as-is” process on the web.
• Project teams began asking their project managers Level 2 questions.
• The OD group continued out of process, stepping up process improvement work-effort to dangerous levels (above 6%) with undetermined results.
Cycle 2: Level 2 Process and Infrastructure Updates
The goal of Cycle 2 was to fill the gaps between existing “as-is” processes and CMM Maturity Level 2 requirements. Training Program and Peer Review competencies defined at CMM Level 3 were also integrated into the Cycle 2 workload. Teams continued evaluating their processes in weekly increments according to life-cycle phases, trusting to the CMM advisor and their facilitators to advise them of process shortcomings. As the teams defined process solution-sets, the CMM advisor verified that proposed processes changes were CMM Level 2 compliant. Facilitators validated that the processes changes could be implemented within the context of the organization culture and technology framework.
Each week, changes to “as-is” processes were socialized with project staff, who tested them against cultural and technical realities. Feedback was collected and relayed to respective teams. As in Cycle 1,100% participation and 4–6% organization work-effort was requested. The PMO tracked progress and posted metrics on the organization web site.
Midway through Cycle 2, with only three months remaining in the CMM attainment initiative, the director that had opted for an OD transformation approach acknowledged that his team had fallen behind schedule. Unsure what had gone wrong and with no clear plan for catching up with the other teams, the OD team requested that the Governance Board redesign their program. The team reorganized into eight Fixed Focus Groups (FFG), each charged with a phase in the software engineering life cycle. The goal of each FFG was to fill the gaps in the “as is” processes with the minimum changes required to satisfy CMM Level 2. In the rush to meet corporate attainment objectives, project staff feedback was not solicited nor were new processes tested until the end of Cycle 2.
With Cycle 2 over, project teams took a well-earned breather from weekly process meetings. While the teams rested, facilitators and process engineers continued their CMM attainment activities:
• Process engineers updated process specifications.
• Project teams complete project plans, software development plans, configuration management plans and quality assurance plans using templates designed by facilitators and tested by the Pilot team. They also created binders of software development work products in anticipation of corporate assessment.
• Corporate assessment teams formally assessed key projects against CMM Level 2. Of the 17 organization projects, five mission critical projects were selected for assessment.
• Facilitators prepared a gap analysis by life-cycle phase between the new processes and CMM Level 3 competencies and developed templates for the Organization Training Plan, Intergroup Coordination Plan, and Level 3 updates to the Software Quality Assurance Plan, Software Configuration Management Plan, and Software Development Plan. They also prepared workshop material for the Cycle 3 Rollout.
• The organization structure changed, transforming the Governance Board into a fully functioning Software Engineering Process Team (SEPT). A SEPT manager, a quality assurance auditor, and a training coordinator were brought into the organization.
• The Level 3 Pilot team (operating on a different cycle schedule from the rest of the organization) consolidated Project Software Development Processes (PDSP) from each team into an Organization Standard Software Process (OSSP) and wrote tailoring guidelines. When the OSSP was complete, the Pilot team disbanded.
Exhibit 2. CMM Assessment Performance Organization vs. Corporation
• The PMO updated the metrics database and stored Level 2 compliant process on the web.
• Project teams started asking Level 3 questions of their project managers.
• The Best Practices group prepared to rollout 20 culturally compliant software engineering procedures. The team elected one member to orient the rest of the organization on the best practices suites and disbanded, their work accomplished.
• The OD group rushed to complete their Level 2 processes and plans and to reverse engineer work products prior to assessment.
Despite constant guidance by the CMM Advisor, corporate assessors identified Level 2 defects in project work products of all teams. Most notable among these was the lack of configuration management discipline and weaknesses in project estimating techniques and requirements traceability. Projects were asked to create action plans and were given six weeks to execute them. All teams, including the OD team, completed their action plans within four weeks.
The number of partially compliant common features ranged from as few to as four to as many as 14. From the perspective of the corporation, results were unexpectedly good. As Exhibit 2 shows, the organization outperformed other corporate centers that had been working on attainment activities for more than nine months.
Cycle 3: Level 3 Rollout
By the start of Cycle 3, a continuous process improvement mentality was evident in all teams except the OD team. Project managers knew how to design, implement and rollout process changes as required. They understood the cultural factors that make the implementation of new processes succeed and those that make new process implementation efforts fail. Because continual coaching and process engineering were no longer needed, the team workshop concept used for Cycle 1 and 2 was abandoned in Cycle 3.
The management team empowered the SEPT to control and manage organization process improvement activities. The SEPT structured the Level 3 rollout as a series of five workshops. Held once a week, these workshops introduced process improvements required for each of the remaining Level 3 KPAs. All process team members attended these meetings. The teams returned from the workshops with tools, templates and implementation instructions. Completing the templates and implementing processes and procedures introduced in the workshops would complete the implementation of the Level 3 infrastructure. Teams leads were given 12 weeks to complete Level 3 related action plans and prepare their teams for a CBA-IPI. With no more work to do, facilitators and most of the PEs left the project at the end of Cycle 3.
Results
In 28 weeks, this organization using a Rapid CMM Attainment Technique (R-CAT) went from “last out of the gate” to an organization with a Level 3 infrastructure. By doing so, they outperformed many other development centers in the company. Because parallel attainment approaches were used in the organization and across the company, the performance of R-CAT against Process Engineering and Organization Development techniques can be demonstrated:
• R-CAT helped this organization of 350 people implement a Level 3 infrastructure within 28 weeks. At the end of the initiative, the prospect of the organization completing a CBA-IPI at Maturity Level 3 was good.
• Process Engineering techniques used by the Level 3 Pilot team required 18 months for a team of 20 software engineers to construct a Level 2 process infrastructure.
• An Organization Development approach threw one team off track within the first eight-week cycle. The team scrambled to prepare for assessment and was unable to implemented Level 2 discipline at the staff level within the 28-week CMM attainment initiative.
• Other development centers in the organization, using a variety of attainment strategies delivered by a variety of industry respected consulting firms fared worse. Eighteen months after start of the corporate CMM attainment effort, the probability of these organizations attaining Level 2 compliance on a CBA-IPI was believed by corporate leadership to be poor.
Unfortunately, due to poor preliminary assessment results in other organizations, corporate leadership lost interest in the CMM attainment program shortly after this initiative was complete. Without corporate leadership commitment to process improvement the organization may ultimately abandon the process improvement framework it put in place during the initiative.
Summary of Method
R-CAT clearly outperformed other CMM attainment techniques in this organization. The differentiators in the method can be summarized as follows:
• The entire organization is trained in a rules-based model of organization behavior and is engaged in the attainment process.
• The structure of the attainment framework is an interlocking combination of teams, facilitation, process engineering, and governance.
• Iterative attainment cycles form the foundation of transformation activities; rest breaks are planned between cycles to allow the organization to assimilate change.
• Transformation activities are “chunked” into logical increments for sustainable delivery.
• Staff participation and work effort are tracked by the PMO and maintained within close tolerances.
• Process engineering against an ideal or academic model is not allowed.
• All process changes are judged to be culturally acceptable before they are integrated into the “as-is” process baseline.
• Consults perform non-project related process improvement tasks such as creation of templates, gap-analyses, process flow and specs, thus freeing project staff to do the “real work” of software engineering.