You wanted WHAT!? A systems engineering case study
Release 2.0 of ibm.com’s Integrated Solution Marketplace (ISM) was one of the first projects where IBM Global Services implemented their Systems Engineering and Architecture (SE&A) methodology. The ISM management team embraced the SE&A methodology, where requirements analysis is a core project driver, to help align stakeholder expectations and to deliver a feasible solution within the defined project schedule and budgetary constraints.
The SE&A methodology emphasizes clear definition and understanding of stakeholder requirements in the initial project phases. Subsequently, system requirements are derived from these stakeholder requirements, and further decomposed and allocated to well-defined components that become the building blocks of the solution and verification architecture.
This gives the project team a powerful instrument to track progress, assess, locate and mitigate risk, and assess the impact of change requests.
The results of utilizing SE&A for ISM 2.0 were positive. Consensus on core stakeholder expectations was set during the SE&A reviews and the project delivered on these expectations. In addition, the planned SE&A reviews allowed the team to identify and resolve key project issues before proceeding into subsequent project phases. This discipline, along with the attention to detail in developing quality requirements resulted in greatly reduced re-work for the project. Qualify phase testing reported one tenth of the anticipated defects projected and the quality of the deployed application was further substantiated by post launch defect reports. Besides producing a quality application that met stakeholder expectations, ISM 2.0 was delivered on schedule and five percent under budget.
ISM Project Background
Two trends in the IT industry in late 1990s led to the development of ISM. The first was the exponential growth of Internet usage and its increasing importance as a marketing and sales channel. Second, IT customers increasingly requested complete solutions to solve their business needs rather than discrete software, hardware and service elements that they had to integrate on their own.
IBM was an early adopter of the Internet as a communication channel, but their web presence was discrete and product oriented. IBM and its business partners realized that they were losing valuable leads and potential customers because of this product-centric approach.
ISM was initiated to provide ibm.com users with a single point-of-entry for solutions-oriented browsing for integrated services, technologies, and products. The goal was not to replace the traditional face-to-face sales strategies, but rather to enhance them by providing initial information to potential customers and to capture valuable leads for the sales organization.
Organizational Context: Divergent Requirements from Stakeholders
The ISM project resided within the ibm.com line of business and interfaced with almost every other IBM line of business. IBM Global Services performed the technical implementation of ISM. The ambitions for the project were high, reflecting its key role in IBM’s strategy to become a solutions provider. Different lines of businesses, each with its own business processes, had diverging requirements and expectations for ISM. ISM imposed requirements and constraints back onto the lines of business, as well. Information structures had to be aligned for presenting products and services as part of an integrated solution. The different business processes had to be aligned to enable an integrated, “single point of contact” sales approach. It was a challenge for the ISM team to obtain concurrence on ISM scope and intent from the numerous stakeholders. As one team member put it, “Everyone wanted everything.”
ISM – First Release Deployment
The development team for the first release of ISM had difficulties in pinning down the requirements in this environment. Ultimately, the management team had no choice but to cut function to the bone to have a chance to reach the scheduled deployment date. When ISM 1.0 was deployed a few weeks late, it had only a fraction of the function originally envisioned.
ISM – Preparing for the Deployment of the Second Release
The second release of ISM (Version 2.0) had to implement much of the capability originally envisioned for ISM 1.0. Jim, a senior project manager within the ibm.com organization, was appointed Product Development Team Lead (PDTL) for ISM 2.0 in the winter of 2002. The role of ISM in IBM’s solutions strategy resulted in increased executive management attention to the ISM project. Jim knew he needed a dedicated team with complete focus on a realistic and stable target. As he put it, “We had to drive a stake deep in the ground with regard to project scope.” Jim’s approach was to “…focus the team on clearly articulated and achievable objectives, exercise the right amount of process and discipline, push decision making down to the lowest level possible, and promote accountability…” The team was strengthened with the addition of two experienced, IBM certified project managers.
An explicit decision was made to follow a “time box” approach due to the importance of getting critical function deployed by year end. A key objective at the start was to manage executive expectations. It was important to articulate what was achievable within schedule and cost constraints, and then get this agreed to by management.
Following Jim’s philosophy of “clearly stated objectives,” one of the first project tasks was to baseline ISM requirements. Jan, the requirements-manager on the ISM team, consolidated information from different sources, identified the requirements and captured them in a Lotus Notes repository. Formal and explicit documentation allowed prioritization, trade offs, and enhanced communication. As part of managing expectations, Jim had to get stakeholders to prioritize their requirements. In the absence of forcing this issue “…all their requirements will have a priority of one.”
Deployment of the Formal IBM SE&A Methodology
Ted, the Solution Project Manager for ISM from IGS, recognized the convergence between Jim’s approach and IBM’s Systems Engineering and Architecture (SE&A) methodology. In late winter 2002, when ISM 2.0 was well into the concept phase, he contacted Paul and asked him to get involved with ISM.
Paul is a business area manager in the SE&A organization within IGS. One of his team members, Vito, an experienced system architect, had already undertaken training in the SE&A methodology and had been exposed to ISM as part of the ISM 1.0 architecture review process. Paul and Vito presented the SE&A approach to Jim in a teleconference. Paul recalled how Jim had asked some tough questions indicating an immediate understanding of the concepts. From Jim’s perspective, SE&A offered the well-founded methodology and process-rigor that he needed to control project scope. As he put it, “These guys spoke my language.” He embraced the SE&A approach proposed by Paul and Vito.
For Paul, this was a very welcome reaction. Without significant and proactive management commitment, SE&A can become yet another process on top of all the others. “A process without passion,” as Vito put it, “can make any process ineffective.” Fortunately for ISM 2.0, the SE&A process mandated by the leadership team was well received by most core team members and stakeholders.
IBM’s SE&A Methodology – Key Drivers and Concepts
At its core, Systems Engineering seeks to facilitate a thorough understanding of a need or deficiency, and then to provide a framework to develop a solution and verification architecture to resolve the need or deficiency. This requires up-front investment to:
- Ensure a common and consistent understanding of the need to be satisfied and the capabilities to be developed and integrated between the stakeholders and the project team.
- Translate this understanding into a robust solution and verification architecture to serve as a foundation for detailed design, development and qualification of the solution.
Avinash, Jim’s project manager, responsible for managing IGS as a supplier to ibm.com, commented, “Investing up front makes intuitive sense but it is rarely done since the results show up much later.”
The SE&A methodology is characterized by:
- Making a clear distinction between the problem domain (mission and stakeholder requirements) and the solution domain (system requirements)
- Clearly defined stages in the development cycle with respect to the level of detail with expected maturity of the solution and documentation
- Intense focus on getting requirements right by defining their final acceptance criteria for compliance
- A solid architecture and high level design that is driven by and compliant with the requirements
- A common, structured, traceable and richly attributed repository for requirements and design information
- Structured and scored reviews to ensure the quality of work products at each stage in the project
- A high level of interdisciplinary collaboration to ensure that all aspects of a problem or decision are accounted for
- An SE&A team charter that includes overall responsibility for the technical solution and serves as the program or project manager’s technical resource while interfacing with the stakeholders, the development team, and the verification (test) team.
Development of the Second Release (ISM 2.0)
Getting a Firm Grip on ISM 2.0 Project Scope and Requirements
Vito joined the ISM 2.0 team as Lead Systems Engineer in April 2002. He was responsible for all SE work-products and deliverables. Vito had 10 days to prepare for the ISM 2.0 System Requirements Review (SRR). At this review, IGS had to demonstrate to ibm.com an understanding of the required ISM capabilities represented by a complete set of system requirements. Further, the SE&A team had to demonstrate a feasible concept for implementing the technical solution within cost and time constraints.
This would have been impossible without the work already done by Jan. While requirements had been identified and the fundamental architecture formulated, there were outstanding issues to be addressed before the team had stakeholder and system requirements ready for the systems requirements review.
Vito organized several sessions between architects and stakeholders from ibm.com to prepare for the SRR. Ultimately, the number of stakeholder requirements was reduced from the original 100 (identified by Jan) to 46. Rather than signifying a reduction in scope, this reflected an effort to achieve the correct level of detail and consistency in the requirements.
The rigor in defining stakeholder and system requirements was new to the team. Jan recalled her experience from earlier projects. “We used to give our requirements to the development team and they would disappear behind a curtain, and we did not see them again before they returned with the code that might or might not have done what we needed.” Keith estimated that they spent about 5 to 10 times as much time on requirements as what he was used to. “It was a tedious and painstaking process, but it definitely paid off in the end.” Jeannette, the Lead Architect, looked back at the SRR as one of the most valuable project activities. The collaborative review sessions resulted in a single, agreed-to set of system requirements. This proved to be key for the phases that followed by ensuring what was being designed, developed and tested met the business expectations. From a project management standpoint, Avinash was thrilled. “The fact that we could count the requirements provided us with a very strong project management metric.”
Despite the successful SRR, Vito was not completely satisfied with the requirements structure. He felt he lacked a means to keep the primary project objectives in focus. Thinking about his SE&A training, Vito recalled the concept of Critical Mission Requirements, which capture the overall project objectives in a small number of clearly articulated statements.
The ISM team agreed to five Critical Mission Requirements. These proved to be invaluable in managing the project scope during subsequent project phases.
The ISM 2.0 Project Infrastructure
The ISM 2.0 team was highly distributed geographically. Jim, working from a “home office” like many of the ISM 2.0 team members, was a strong believer in the concept of “virtual teams.” He utilized a Lotus Notes Team Room and teleconferences very effectively. Apart from one interdisciplinary, co-located session that is discussed below, every ISM 2.0 meeting and review took place on-line to avoid travel time and expenses.
Jan had already captured requirements in the Team Room repository, but this repository lacked the basic features of a requirements management (RM) tool. Vito had previous experience as a Lotus Notes developer and customized the Team Room environment to accommodate some basic RM function. Traceability management and other RM function was provided by a specialized RM tool, Rational Requisite Pro.
Defining the High Level ISM 2.0 Design
Mapped onto the traditional IBM project framework, a completed SRR marked the end of the Concept Phase. The project now entered Plan Phase, focused on finalizing the architecture, high-level design and component requirements, and preparing for the Preliminary Design Review (PDR). Ted, the ISM Solution Project Manager, would then commit to scope, cost and schedule.
Typically in IGS, the system architecture and the high-level design are driven and defined by the software development team, while the project manager supervises customer communications and expectations. Ted recalled from his earlier projects, and from ISM 1.0 in particular, that he had to engage the customer with detailed discussions focused on project scope. He found this frustrating, as it drew his attention away from managing the project and engaging the customer at a business level. Ted found that having the SE&A team responsible for the technical solution on ISM 2.0 eliminated this frustration.
The elevated attention and visibility devoted to the system architecture and high-level design on ISM 2.0 were new to the team. Likewise, the PDR to ensure compliance with the stakeholder and system requirements before the project could proceed was new. For ISM 2.0, the high-level design and architecture formulation activities were strictly requirements driven. Component level requirements were derived from system requirements to define the contribution of components to the overall system functionality. Traceability back to stakeholder and originating system requirements was established and rigorously managed in accordance with a traceability map.
As the detailed schedule was worked out, it became clear that all the stakeholder requirements that were agreed to during SRR could not be implemented in the desired timeframe. The team re-engaged with stakeholders and reviewed each stakeholder requirement to assess its contribution to Critical Mission Requirements. Jan recalls, “This killed discussions about this whistle or that bell and focused us on the core functionality to be implemented.” The number of stakeholder requirements to be implemented as part of ISM 2.0 was reduced to 35, without compromising fundamental project scope as stated in the Critical Mission Requirements.
The SE&A team facilitated the discussions with stakeholders and provided a rational and structured approach for evaluating requirements. This, combined with robust documentation and analysis at every step avoided potentially heated and contentious debates.
Change management was given particular attention. Traceability allowed the team to immediately understand which components would be affected and what the impact would be to the schedule and budget.
If issues surfaced at the implementation level, traceability enabled the team to assess system level impacts. If an issue had the potential of affecting a stakeholder or system requirement, the stakeholder who owned it was involved in issue resolution.
The next challenge for the ISM team was to identify component level requirements and to produce a high level design in support of these requirements. These were to be reviewed at the Preliminary Design Review (PDR) which was scheduled for mid July 2002.
Issues identified during PDR emphasized the value of the scored reviews. Every member of the project management team would highlight the value in being able to quantify the quality of the work products and to be able to assess their maturity and progress towards the defined and agreed upon project goals.
ISM – Detailed Design and Development
One of the actions taken to resolve the issues identified during the PDR was to bring the geographically dispersed SE&A, development and business teams together for a week long series of Joint Application Development (JAD) sessions. The JAD sessions produced an action plan which allowed the team to resolve all key issues identified during the PDR and to proceed on schedule towards the next key check point, the Critical Design Review (CDR).
In preparation for the CDR the detailed ISM 2.0 design was documented in a series of Internal Design Documents (IDDs). Each IDD documented the implementation of one or more stakeholder requirements.
The CDR was conducted in late August 2002. This review involved selected stakeholders, which some development team members questioned. However, from an SE&A perspective, CDR is an important review, as it represents the last opportunity stakeholders have to identify and clarify any issues before full-scale coding begins.
Another concern expressed by the development team was that they had only five weeks to finalize the code and conduct unit testing before the Test Readiness Review (TRR). But the level of detailed design specifications allowed the management team to allocate additional development resources to the project to handle specific design elements.
ISM - System Integration Testing and Delivery
When testing began, several people started to get worried. The test team was reporting very few defects. “At the start, this would be normal,” Vito commented, “but as the tests got going, defects would normally start piling up.” Kelvin, the test lead, was reporting a two percent defect rate, though he had projected a rate of about 20%, based on experience with previous projects of similar size and complexity. Exhibit 1 below shows actual versus anticipated defects during functional verification test. Kelvin commented that he could not recall ever having seen code of this quality.
The groundwork for smooth testing was done long before TRR. Well-documented and traced stakeholder requirements and acceptance criteria formed a solid basis for developing the functional verification test cases. Kelvin pointed out that close collaboration with the architecture team during test planning was a key factor in the low number of defects. The high level test acceptance criteria and test architecture had been reviewed at PDR. Subsequently, the architects engaged directly with the test team to define and review test cases. Through this close interaction potential misunderstandings were addressed, and the test team gained a deeper understanding of the architecture and the functions of the application. According to Kelvin, the result was “flawless test cases.” Kelvin had estimated 6-7 weeks for testing. It took five. User acceptance testing, according to Keith, took a matter of hours, rather than weeks as would normally be expected.
ISM 2.0 went live on the ibm.com website in December 2002, slightly earlier than planned. At project completion, spending was approximately five percent under budget. When Ted was asked to characterize the rollout of ISM 2.0 he summarized it in one word, “Uneventful.”
Exhibit 1 Anticipated vs. actual defects found during functional verification test
The deployment of a new application or significant new release is often associated with significant problems during the initial operation. Exhibit 2 shows the extraordinarily low total defect count over the whole project and the first three full months of operation.
Exhibit 2 Total defects reported
Observations; ISM vs. Non SE Projects
Exhibit 3 below shows how ISM compares with 19 other IBM projects that did not utilize SE. We see that the majority of the ISM 2.0 defects were captured during the plan phase; when they can be fixed at a low cost with no significant consequences to the schedule. In the non-SE&A projects however, the bulk of the defects are identified during test where the rework has a significantly higher price.
Exhibit 3 Total number of defects detected by phase
Exhibit 4 below depicts costs to fix defects detected based on the defects identified by phase. Recent studies estimate that the cost to fix a defect found in concept phase of a project is approximately $20 per defect while the cost to remedy a defect found in the deploy phase of a project will be $1,080. Utilizing this criteria, ISM demonstrated 2.4 times less cost to fix defects than that of projects not utilizing SE.
Exhibit 4 Estimated costs to fix defects by phase for ISM vs. projects not utilizing SE
Summary and Conclusions of this Case Study
The SE&A methodology for ISM 2.0 effectively addressed the concerns of the ISM management team regarding project scope and focus. The requirements-driven SE&A process accomplished Jim’s objective and “…focused the team on clearly articulated and achievable objectives.” ISM 2.0 was delivered on schedule and under budget in full compliance with the set of requirements established for the project. In summary, SE&A enabled the ISM 2.0 team to:
▪ Gain and Maintain Consensus on Scope and Objectives: The definition of Critical Mission Requirements allowed the team to focus efforts on achieving project objectives. This approach allowed for removal of lower priority functionality without compromising on the overall mission for ISM 2.0.
▪ Maintain a Solid Requirements Baseline: The stakeholders were involved in the development process. This ensured a solid requirements base under unyielding change control that was understood and agreed to by everyone involved. Traceability allowed for detailed assessment of change requests, which prevented scope creep and facilitated sound trade-off studies and conflict resolution.
▪ Keep the Project on Course: The methodical way the SE&A team interacted with the stakeholders on a technical level gave the program and project managers the confidence to “let go” of detailed involvement in scope and architecture discussions and freed their time and attention to focus on their management and leadership tasks. The SE&A checkpoint reviews kept all project participant expectations aligned and addressed any identified issues prior to moving on to the next phase.
▪ Assert Better Project Control: The scored SE&A reviews not only measured the quality of the work products themselves, but their compliance with the objectives of the project. This, combined with the comprehensive requirements and design repository, allowed the project management team to identify, locate and act on risks at a much earlier stage and with much higher fidelity than from previous projects. As Jim put it when he summed up the benefits of SE&A: “The project management game is all about risk. I felt like I was in control of the project, rather than being controlled by it. SE&A not only lowered the risk, but it gave a better understanding of where the risks were.”
▪ Minimize Rework: The emphasis on up front requirements definition and design provided a solid framework for detailed design and coding and a low rate of defects and rework after the development phase, as seen in Exhibit 2.
▪ Deliver a Quality Product: The unexpectedly low defect rate during system level testing and initial operation indicated an exceptionally high quality of the code.
▪ Deliver on Schedule and Under Budget: ISM 2.0 went live on the day committed and 5% under budget.
An IT project that delivers on every committed requirement on time and under budget is still an exception, rather than the norm. Given that ISM 2.0 was the first end-to-end deployment of the SE&A methodology, even the most optimistic were surprised by this degree of success. A project team rarely welcomes being a “guinea pig” for new processes, and a certain level of dissipation could be expected. In this case, leadership understood the SE&A potential and embraced it 100%. The team, which Jim referred to as “an incredible stable of talents,” was receptive to the new approach. According to Vito, “The requirements might have been the project glue, but it was the team that made it stick.”
This paper is based on the original Case Study written by Dinesh Verma, Eirik Hole and Vito Vitale. It has been edited and reformatted in order to meet the requirements of the PMI Global Congress.
Acknowledgements of the original authors
This Case Study was made possible through valuable input from Jim De Piante, Jan Van Hoomissen, Keith Instone and Avinash Kohirkar from ibm.com, and from Pierre Carlson, Jeannette Decker, Ted Hamilton , Paul Popick and Kelvin Wong from IBM Global Services. Thank you for sharing your thoughts and experiences
© 2006, Jim De Piante
This version originally published as part of the 2006 PMI Global Congress Proceedings – Seattle, Washington, USA