Simulation and visualization of performance across subsystems in complex aerospace projects
Today's aerospace products are more complex, while in addition are developed by teams located across the globe. The challenge is to optimally organize, direct, and manage these teams and their interactions, dependencies, and priorities during the program. The complexity of these systems and multiple layers of system architecture make it difficult for internal and external teams to develop and maintain a shared situational awareness of the total product architecture, project schedule, and most significant risks. Traditional detailed work breakdown structure (WBS) and earned value management approaches often exacerbate rather than help performance under such complexity. In this paper, we show a simulation and visualization approach to promote shared situational awareness. Sustainable, visual tools allow teams to keeps their focus on real progress, feasible and valuable coordination activity, and systemic product risk throughout.
Situation: The Nature of Complex Aerospace Projects
Aerospace product development as an activity has been understood well as the integration and management of a system of systems. This systems nature of the product itself has not changed recently, yet the design content and complexity of subsystems have continued to increase. More significantly, recent decades have shown a dramatic transformation of how these systems are developed and the way project organizations are engaged around these subsystems.
Most commonly, in the past teams working on a complex aerospace system were located in the same buildings, corporation, and work culture. The development phases were more likely to have been sequential, and the workforce involved likely to be experienced with the previous generation of product, and relationships—both formal and informal—existed among teams as they worked together over time and across multiple projects. In contrast, recent aerospace development projects are marked by meaningful changes in how product, process, and organization architectures are organized and linked:
- Multiple layers of system and subsystem, with substantive design responsibility layers down
- Complex dependencies fall across subsystems owned by different teams
- Dispersed engineering teams and supply chain with less chance of a shared, common background
- Pressure to proceed in a dramatically concurrent fashion, increasing risk of rework, quality, and delay
Difficulty: A Perfect Storm Leading to the Decline of Judgment
These changes to both the systems being developed and how teams are organized to participate in development have led to a “perfect storm” combination of difficulties. This perfect storm is characterized by the thinning of the workforce bench, variation of work practices globally, impact of dependencies across major subsystems, inflexible safety requirements that drive huge rework and risk, and the high cost of communication.
Thinning of the Work Force
A recent study of defense and senior industry executives (National Research Council, 2001) indicates that the defense industry is able to hire adequate numbers of engineers at the current work level and in all likelihood will continue to be able to do so in the near future. The real concern is whether the quality of these engineers will deteriorate. The Cold War provided work opportunities that were attractive to engineers in the defense industry. Current cutbacks in defense programs and uncertainties about the future have made this sector less attractive to the engineering profession. Increased competition and the need for engineers in other economic sectors such as information systems and biotechnology are providing engineers with many new opportunities.
These concerns for the prospects of a talented, well-educated, highly motivated, and appropriately experienced defense aerospace science and engineering work force were based on four factors:
- Recent reductions in defense projects combined with reductions in hiring have changed the age-experience composition of the defense aerospace work force.
- The reduction in projects means a decline in new starts, hence a decline in opportunities for design experience. This leads highly qualified technical workers to consider the defense aerospace field less desirable. The lack of opportunities makes it much more difficult to attract and retain top talent and to build and maintain the necessary experience base.
- The overall decline in Department of Defense funding has increased unit weapon system cost; therefore, scientists and engineers will work on even fewer projects in their lifetimes and thus will have less experience across a broad spectrum of technologies.
- The increase in unit cost could lead to gaps of years, perhaps, between the developments of new systems; in the interim, specifically trained and experienced workers may be lost.
Variation in Work Practices Globally
The many variations in global work practices range from the obvious to the complex. (Johri, 2008) describes how geographically distributed work has been around for centuries and that advances in technology have reshaped the extent and nature of distributed work; an increasing number of people now work as members of global or virtual teams with co-workers who are dispersed across states, countries, and continents. Some of these changes have had contradictory affects. For example, on one hand, they have brought people together to solve significant problems, and, on the other hand, they have led to increased interpersonal and organizational breakdowns. Johri states that geographically dispersed workers lack mutual knowledge and common ground, leading to misattributions and breakdowns in communication and collaboration, greater interpersonal conflict, and lack of trust; they are prone to subgroup formation and ethnocentrism and find it harder to share knowledge and expertise.
Several NAE studies (National Academy of Engineering, 2004) have examined the needs of globally distributed work forces and have pointed out key areas that can affect the ability of global teams to collaborate effectively in developing complex systems. These needs and concerns can be described as follows.
- Technical: Ability to think mathematically; sound knowledge of appropriate basic science; good knowledge of a specific discipline; maintenance of current knowledge and practice
- Personal: Ability and willingness to learn: appreciation of limits to knowledge; good communication skills: appreciation of international dimensions
- Professional: Commitment to high standards; appreciation of personal and ethical responsibilities; ability to handle uncertainty; ability to communicate effectively; ability to communicate effectively in more than one language including English
- Managerial: Ability to work in a team;, appreciation of management concepts and issues; ability to lead and manage personal, financial, and technical resources
Impact of Dependencies Across Major Subsystems
A recent and visible example of this was the Boeing-led development of the 787. Even early on, the combination of new technologies and concurrent yet massively dispersed team responsibilities caused some to be concerned about the feasibility of early cost and schedule projections for the aircraft. A BusinessWeek article in mid-2006, before the multiple delays now announced, reported this concern:
“Technical glitches, missed deadlines, and stretched nerves are par for the course with new planes. But far more than a new plane is at stake. Boeing has undertaken a grand business experiment with the Dreamliner. In a bid to tap the best talent and hold down costs, the aerospace icon has engaged in extreme outsourcing, leaving it highly dependent on a far-flung supply chain that includes 43 “top-tier” suppliers on three continents. It is the first time Boeing has ever outsourced the most critical areas of the plane, the wing and the fuselage. About 80% of the Dreamliner is being fabricated by outside suppliers, vs. 51% for existing Boeing planes… The Dreamliner's mounting challenges call into question whether such a radical business model can succeed, and whether the advantages of collaboration on such a scale are outweighed by the loss of logistical and design control.” (Holmes, 2006)
Inflexible Safety Requirements Drive Huge Rework and Risk
Insistence upon safety of systems and subsystems, whether hardware or software, based on accepted rigid and inflexible standards can lead to unplanned rework and eventual risk associated with these standards. Safety-critical systems and subsystems must be engineered to meet system safety requirements, and there are many system safety standards that are used for the “design for safety” exercise. There are two kinds of safety requirements: process oriented and technical. Both need to be addressed and properly documented within a program, project, or facility. Process-oriented requirements determine what needs to be done to ensure system safety. Technical requirements are those that specify what the system must include or implement (e.g., two-fault tolerance). For example, over the past several years, a great amount of experience with mission critical software safety requirements has been gained. Considering legacy safety-critical computer systems, it is possible that software safety requirements may not have been formally specified during development. In the event that process-oriented software safety requirements are imposed on a legacy system after the fact, where software development features don't exist or are incomplete, the question becomes “what should be done and how can this be accomplished?” Certain risks are associated with only meeting certain software safety requirements in a legacy safety-critical computer system, and these risks must be addressed should such systems be selected as candidates for reuse. Experienced software and hardware systems engineers have been able to make informed decisions as to how to address the question of maintaining safety in mission-critical systems in the case of missing or incomplete standards. Lack of this experience could necessitate a large amount of rework to compensate for or overcome perceived potential safety exposure—the other side of this issue is to ignore or implement expedient safety features, leading to inherent risk.
Cost of Coordination is High
Engineering systems development is dealing with larger and larger scales and increasing complexity, with systems lifetimes often exceeding those of the designers. Such systems, because of their size and complexity, require human or social considerations that must be accounted for in the design process and eventual project design and management. Modern aerospace design must therefore be centrally concerned with human cognitive abilities and limits, as these often represent some of the most difficult parts of the design problem. One of the most central and important aspects of complex systems development is coordination, the interaction of teams to satisfy dependence.
A 2004 National Institute of Science and Technology report about engineering infrastructure claimed that 40% of engineering time is spent locating and validating engineering information. Furthermore, 30% of project costs are wasted due to poor communication between systems leading to a loss of US$15.8 billion annually. Although the NIST report identified areas where savings can be made—namely, reuse of information—project delivery times can be reduced up to 50%. Most of the NIST report's data referred to US-based projects. For globally distributed teams working on complex engineering systems, the situation is exacerbated due to reasons outlined above. Value-based coordination management is a key ability for global teams to realize the gains described by the NIST study. The cost of failing to provide effective coordination can easily lead to serious project consequences, including significant schedule slip, cost overruns, and project cancellation. Described later in this paper, one of the key root causes for a failed complex reconnaissance satellite project was traced to a lack of continuous coordination; before cancellation, progress on the project was years behind and billions in cost overruns.
Our Approach: Visualization and Simulation of Project Across Subsystems
Our team has been innovating and deploying an approach to overcome the decline in judgment, inaccurate forecasts, and subsequent poor performance in complex engineering projects. The approach combines visual modeling and simulation delivered as a collaborative design exercise. Teams iterate around alternate designs of their project architecture, evaluating the value and feasibility of each with simulation-based forecasts.
Judgment for Teams Through Architectural Project Design
Our work in complex product development project modeling began in 1995 at the University of Tokyo and MIT. Our first goal was to capture the essence of complexities and dependencies in systems development by global teams. (Moser, Mori, Suzuki, & Kimura, 1997). We focused on the need to maintain an architectural, visual view of the total system across product, process, and organization points of view.
In the past, the activity of product development was sufficiently consistent over the career of an engineer that this architectural view became embedded in their professional judgement. To overcome the recent lack of such judgment, the visualization and simulation becomes a learning-centered planning exercise through collaborative modeling. As the various stakeholders participate in the design exercise, the interplay of architectures, disparate priorities, and the subsequent feasibility of delivering value are highlighted. This design dialogue happens early, informed by architectural project prototypes, and promotes accelerated convergence on shared objectives and options for proceeding.
We are impressed by recent discussion of product architecture in the Value-Based Architecture Selection approach proposed by (Cameron B., Catanzaro S., & Crawley E., 2006) for NASA. What is missing in their discussion is the project and organizational aspects required to make estimates of what is eventually feasible, timely, and affordable. Since 2000, we have established a proprietary set of tools, called TeamPort, which is used by stakeholders and team leaders in project design sessions. Two examples of the same project are shown in TeamPort (Figs. 1 and 2). We are confident that our project design techniques are complementary and therefore a combination of the project design practice in concert with value-based architecture selection has merit.
Forecast Accuracy Through Converging Iteration
As a project design exercise proceeds, the relevant elements in a project model and forecasts of likely schedule, cost, and risks improve. In addition to calculating the overall expected coordination activity, teams are stimulated to understand when, where, and why coordination occurs, and how lack of coordination will potentially cause systemic, propagating delay, rework, and poor quality.
These project models can be generated early, while total architectural options are still under consideration. Where complexities and dependence fall within the activities of teams, further granularity in a project model has less value than modeling of the dependencies that fall across team architecture boundaries, and thus drive coordination across teams. It is these coordination activities that are least likely to be predicted based on previous judgement, whereas team and subsystem internal coordination is more likely to be consistent with previous experience.
An engaging experience for team leaders who will be held accountable for results and schedule is a key part of project design, leading to a plan that is both accurate and optimal.
This iterative process is similar to well-understood best practices in human activity involving multiple teams in complex missions: practice games for sports teams, field exercises for the military, rehearsal in the performing arts, 3P and “obey” exercises in manufacturing, and prototyping of products in engineering.
Often early mission studies focus on product/mission desirability yet don't sufficiently include feasibility from a project management view. Since these visual project models can be created earlier and more quickly than traditional, detailed WBS plans, program feasibility can be weighed as part of the overall mission strategy.
We present two case studies from aerospace, each of which highlights the nature of complexity, the risks of not coordinating, and the potential for overcoming these challenges.
Global Aircraft Development
A leading aircraft development corporation decided in the early 1990s to take a traditional, co-located, sequential development process and move to a globally dispersed and concurrent approach instead (Moser, 1999). The strategic benefit of involving global partners and reaching delivery early could be gained only if the project was feasible as assumed. However, the early architecture of the development program was decided based on an expectation of well-defined interfaces, partners working at predictable paces, and no significant change in coordination efforts and costs.
Traditional schedulers predicted 5 years to reach first prototype. Our project design technique, including simulation of coordination activities predicted 9 years, and showed where this delay would originate. The actual results showed strong correlation to our forecasts, including clear prediction of which subsystem relationships would cause the most recurring rework.
This particular project revealed a “perfect storm” combination of concurrency, time zones, and dispersed decision making that drove propagating delay and rework. Assumptions from the start complicated matters: that concurrency is always valuable, that outsourcing (partnering) enables concurrence, and that partners should proceed as quickly as possible once specifications were available. These assumptions as embodied in project architecture led to unexpected communication and coordination effort and large amounts of rework.
It would be unreasonable in such projects to suggest arbitrarily to change task order, delay tasks, or to reduce coordination activity. Instead, we argue that an optimal “project design” should:
- Forecast coordination and its impacts ahead of time
- Allocate and manage coordination when and where most valuable
- Promote concurrency only when systemically beneficial
Ultimately these adjustments to the project architecture and schedule change the priorities and flow of relative progress by teams. The teams themselves—who will need to adjust their behavior in order for better results to be achieved—must be involved in the design of an optimal plan. In the words of a key stakeholder at the aircraft integrator, these teams had to “slow-up” or “speed down their work—counterintuitive to their original judgment yet systemically leading to better results.”
One of the recent worst-case examples of a project that exemplifies the above-mentioned problems involves the acquisition by the National Reconnaissance Office (NRO) of several spy satellites. The satellites in question were the Future Imagery Architecture (FIA) and an advanced radar imagery satellite. Due to reasons described briefly here, the FIA was cancelled in September 2005 one year after the first satellite was to have been delivered with an estimated total cost of US$18 billion (Taubman, 2007). The original cost was set at US$5 billion for the first 5 years, with future budget adjustments as required. The FIA is but one of several satellite programs to break down in recent years, and the multiple failures that led to the program's demise revealed weaknesses in the government's ability to manage large, complex contracts.
Robert J Hermann, Director of the NRO, and on whose watch the new satellite was conceived and ordered, has stated that the contract was technically flawed and not executable from the day it was signed. Boeing, the winning contractor, submitted a low bid in spite of acknowledged (within Boeing) large risks associated with never having built such technology before. Boeing was also given the responsibility for monitoring its own work and, at the same time, the NRO, hurt by budget cuts and the loss of seasoned staff members, lacked the expertise to oversee the project effectively.
Much of the FIA breakdown can be attributed to poor information management and coordination over the life cycle of the program. An important example is associated with one of the subsystems, a set of gyroscopes that help adjust the spacecrafts attitude for precision picture taking. The root cause of failure for the subsystem was traced to a subcontractor that had changed its manufacturing process for a crucial part, in turn leading to a subtle but disabling alteration in the metallic structure that went undetected until Boeing discovered it 3 years into the project. The Boeing program manager admitted that there was a breakdown in discipline and systems engineering on the FIA project.
Consistent with the findings of the GAO report, the failure of the FIA project represents the decline of American expertise in systems engineering, the science and art of managing large, complex engineering projects to weigh risks, gauge feasibility, test components, and ensure that the pieces come together smoothly.
The situation faced by the aerospace industry today includes changes in our work products, our workers, and how we work together. Judgment and embedded practices built on decades of traditional work processes and system architectures are losing relevance as new architectures—product, process, and organization—constantly emerge and overlap.
We have argued that techniques that rest on traditional planning significantly misrepresent coordination, real activity requiring effort, time, and cost. For complex development projects, coordination can consume one third to half of teams' attention. Some falsely assume that coordination is a qualitative notion rather than a real activity, and therefore do not recognize the limited capacity of teams to both do their own work and interact with others.
Our approach addresses complex aerospace systems development initiatives with the goal of helping to guarantee on-time and on-budget delivery of required system performance. Our methods and tools for the design of project architectures include simulation of coordination activity and its impacts. Coordination activity should be managed—designed, allocated, prioritized, measured, and adjusted. Project design exposes old assumptions embedded deeply in our processes and professions, building situational awareness as to when and where coordination has value, and where it does not, for the architectures at hand.
Cameron, B., Catanzaro, S., & Crawley, E. (2006). Value based architecture selection, 57th International Astronautical Congress, IAC – 06 - D.3.1.3.
General Accounting Office. (2005, April 28). Defense management: Key elements needed to successfully transform DOD business operations. GAO-05-629T.
General Accounting Office. (2008, March 31). Defense acquisitions: Assessments of selected weapon programs. GAO-08-467SP.
Holmes, S. (2006, June 19). The 787 encounters turbulence. BusinessWeek.
Johri, A. (2008, October 22–25). Boundary spanning knowledge broker: An emerging role in global engineering firm, 978-1-4244-1970-8/08/, 38th ASEE/IEEE Frontiers in Education Conference, IEEE, Saratoga Springs, NY.
Moser, B., Mori, K., Suzuki, H., & Kimura, F. (1997, May). Global product development based on activity models with coordination distance features. Proceedings of the 29th International Seminar on Manufacturing Systems, Osaka, Japan, pp. 161–166.
Moser, B. (1999, March). Understanding coordination, communication and rework in globally distributed projects. Collaborative Product Development—Leveraging Resources for New Product Advantage. International Association of Product Development, Peachtree City, GA.
Moser, B., Woodward, H., & Halpin, J. (2008, October). A pendulum too far: A case for action vs. detail. Proceedings of the 2008 PMI Global Congress, Denver, Colorado, USA.
National Academy of Engineering. (2004). The Engineer of 2020: Visions of engineering in the new century. Washington, DC: National Academies Press.
National Research Council. (2001). Review of the Future of the U.S. Aerospace Infrastructure and Aerospace Engineering Disciplines to Meet the Needs of the Air Force and the Department of Defense. Washington, D.C.: National Academy Press.
Taubman, P. (2007, November 11). Failure to launch: In death of spy satellite program, lofty plans and unrealistic bids. New York Times.
© 2009, Global Project Design
Originally published as a part of PMI 2009 Global Congress Proceedings – Orlando, Florida