Uncertainty analysis for program management
Martin Dean Martin
Air Force Institute of Technology
John O. Lenz
Air Force Systems Command
William L. Glover
A major problem facing program managers responsible for Department of Defense (DOD) weapon systems development programs today is how to effectively predict and ultimately control and manage program cost growths. During the past ten to fifteen years, cost growths have plagued major development programs. There are many tools available for managers to use in estimating program costs, but most of the methods used do not consider the uncertainty associated with the successful completion of a program in any rigorous or formal manner.
The specific inclusion of uncertainty in estimating program costs formed the basis of a recent research effort by the authors while assigned to the Air Force Institute of Technology (AFIT), Wright-Patterson Air Force Base, Ohio.1
The purpose of the research effort was to validate an entropic cost model for use in predicting and controlling the final cost of weapon system development programs in the DOD. The model was originally formulated under Air Force sponsorship at the University of Oklahoma. A brief background will facilitate an understanding of the model.
To understand the entropic cost model, a few points must be introduced relative to the acquisition environment and the characteristics of information.
The Acquisition Environment
There are two phases of a development program which relate to the potential of a cost growth: pre-award and post-award (See Figure 1). For the entropic cost model, the critical point in time is the contract award for a development program.
Figure 1 Contract Life Cycle
During the pre-award phase (before actual award of the contract) the program manager is primarily concerned with influencing future cost growth of his program. To accomplish this task the program manager (PM) has at his disposal certain information which should permit him to structure his decisions in a rigorous manner at the time of contract award. This information includes technical data, cost estimates, and results of risk analysis. Technical data consists of engineering estimates and feasibility studies conducted by either the government or the contractor. Cost estimates are available in four principal forms: Cost Analysis Improvement Group (CAIG) estimates; Independent Cost Estimates (ICE); estimates made by personnel organic to the Systems Program Office (SPO); and finally, estimates in contractor proposals. Uncertainty analysis is relatively new as far as being a formal and integral part of the PM's information base; that is, only recently have serious efforts been made to formalize and structure the process of uncertainty analysis during the development program pre-award phase.
During the post-award phase (See Figure 1), the PM must monitor control systems and act to preclude a program cost growth based on his own expertise and that of his subordinates in the SPO. The information available to the PM has certain characteristics that are central to the fundamental concepts of the en-tropic cost model.
Characteristics of Information
The universe of program information relative to the development of a weapon system is comprised of two subsets: ordered information and information that lacks order. Ordered information relates to factors of the program which appear relatively certain as to their ultimate outcome (See Figure 2). These factors generally form the basis for the target (the theoretically “most likely”) cost of the program at contract award. The information in a program which lacks order relates to aspects of a program with uncertain outcomes and form the basis for cost growth during development and possibly production. A conceptual visualization of the SPO information base is illustrated in Figure 2.
Figure 2 Program Information Base
The lack of order, or uncertainty, in program information forms the foundation for the entropic cost model. In the terminology of set theory, it is the complement of the ordered portion of the information base. For the entropic model, the uncertainty of information is conceptualized as approximately equal to entropy, or the disorder, in the program system of the PM and his information. The concept of entropy has its roots in thermodynamics. Entropy in a physical system is the amount of disorder present in the system due to molecular state changes when heat is applied. This property of disorder was extended to information systems in the development of information and communication theory by Shannon and Weaver. The concept was used in an attempt to explain noise in communication systems.2 An extension of the concept forms the basis for the entropic cost model.
An Entropic Cost Model
The entropic cost model is formulated as follows:
In this model, when order is dominant, the final cost is known. When order is not present, there are multiple program factor outcomes possible, and the final cost is uncertain. Thus, uncertainty implies several possible cost outcomes. The purpose of the model is to estimate final cost by using a quantified expression of the uncertainty in the program. The basic goal of the recent research effort at the Air Force Institute of Technology was to test the validity of the model, using data from an actual development program.
As a consequence of time constraints, a test using several development programs was not possible. Therefore, a single program was chosen as the test medium. The program ultimately selected was the Short Range Attack Missile (SRAM) development. Selection of the SRAM was predicated on the following factors:
(1) The program was recently completed;
(2) In terms of dollars, it was within the scope of the effort;
(3) Personnel involved in the source contractor selection for the SRAM were still at Wright-Patterson AFB.
Validation of the model using the SRAM required a re-creation of the program information environment in existence immediately subsequent to the award of the development contract. A review of program documentation disclosed that the data necessary to reconstruct the environment was contained in the source-selection documentation file. This file was not readily available as a result of the document security required by the source-selection process. An alternative method was required to reconstruct the information base for the test of the entropic cost model.
Reconstruction of the Post-Award Information Environment
The limited access to documentation caused an additional secondary objective to be added to the research. That was, in essence, to develop a method by which the uncertainty of information could be quantified in a structured manner. The method needed to not only satisfy accepted research techniques, but be practically useful, if possible, in the realm of the PM and his decisions.
The work of C. Jackson Grayson in his organization of expert judgment and its application to oil well drilling decisions was known to the researchers.3 The technique rested largely on the use of probability statements as responses to questions of well-drilling experts relative to the potential success of the drilling operation in terms of oil production. Specifically, Grayson requested geologist faced with making decisions relative to the location of oil-bearing formations to develop probability distributions as to the probable success of drilling at a specific location. The geologist could, then, subjectively formulate a risk function based on his experience or could select a “Classical” distribution, such as Normal, Gamma, or Poisson, which seemed to fit his mental pattern. The individual's subjective risk function (distribution) would be derived from a verbal lottery with successive questions to specify quantitative point estimates, which when plotted would give a probability distribution. Later, Grayson extended this technique to derive a group risk preference function. A similar type of question/answer format was perceived to be applicable to the program management environment in DOD.
A somewhat exhaustive review of techniques used to structure the opinions of experts revealed the DELPHI method, developed at the RAND Corporation, as a candidate for application of the Grayson method to the instant research. DELPHI is a method for predicting the probability of future events by polling experts, concerning their subjective evaluation as to event occurrence. Each participant is interrogated individually by means of an interview or questionnaire. The process involves four rounds of interrogation. The results of each round are fed back to the participants in an anonymous manner to eliminate the influence of strong personalities. The goal is to refine and revise the subjective probabilities which are being formulated as a measure of the uncertainty of occurrence for each future, alternative outcome. Generally, the DELPHI technique is normally used in forecasting from the present to the future, with responses in the form of what might happen. As applied to the recollection of SRAM program source-selection panel members, the responses not only expressed what was to happen, but assigned probability statements to the outcome measures of unacceptable, acceptable, and exceptional.
By the controlled interview/feedback/interview cycle central to the DELPHI method, the researchers were able to identify some 19,683 possible SRAM program factor-outcome combinations, and assign a probability to each. The calculation base was the nine (9) factors identified during the DELPHI interrogation; each having three categories of outcomes assigned. Thus, there were 39 or 19,683 possible outcomes or states for the program. By means of the computer, entropy was then calculated using the following formula:
Where, 0 ≤ entropy ≤ 1 and
Pi = probability of the ith program factor out
The entropic value of 0.686 was used to compute an estimate for the total program cost as outlined below:
Subjective Probabilities by Means of DELPHI
The DELPHI method was used to poll the original participants in the SRAM source selection to determine the subjective probabilities associated with the uncertain outcomes expected in the development program for the SRAM. These probabilities were used to calculate the entropy in the program at the time of source selection on a retrospective basis. As related to the validation goal for the model, the findings for the effort are significant.
Findings and Conclusions
The actual total cost for the SRAM development program was $439 million. The estimate for this cost obtained by applying the entropic cost model was $456 million. This estimate was based on encountering the worst possible cost conditions during development. Adjustments based on approved changes which were not contemplated at the time of source selection were made to the final cost data. As a consequence, results of the study indicate that the entropic cost model is a valid predictor of development program cost. The power of the model rests in its ability to readily explain uncertainty in a single measure, entropy. Admittedly, the results of one research effort do not validate the model for general applicability. However, the model does have potential as a cost estimating tool for program managers. Further research applying the model to other developmental programs to determine the extent of the usefulness of the entropy concept is planned.
Another significant finding that was a by-product of the research endeavor was the use of DELPHI in uncertainty analysis. Application of the DELPHI methodology to determine uncertain aspects of a development program provides a structured process by which a PM can use his experts to develop rigorous inputs to assist in making key and significant program decisions as related to cost, time, and performance.
Other Possible Applications
Both the DELPHI technique and the entropic cost model merit consideration for application in various areas external to the defense environment. Planning and control for a large, high-dollar value project which entails a moderate to high degree of uncertainty as related to cost could be managed by application of the methodology at selected decision points over time.
For example, large-scale projects in construction, marketing new products or services, advancing technology and exploration for new deposits of natural resources are a few possible areas for application. The amount of effort expended naturally depends on the magnitude of the project and how much time and money management is willing to invest for information, a measure of the entropy in the information base, and the estimated cost outcome for a specific program. Certainly, the benefits as related to cost control should exceed the cost of administration.
1. William L. Glover and John O. Lenz, A Cost Growth Model For Weapon System Development Programs, Unpublished Master's Thesis (Wright-Patterson Air Force Base, Ohio: Air Force Institute of Technology, 1974), p. 124.
2. Claude E. Shannon and Warren Weaver, The Mathematical Theory of Communication (Urbana, Illinois: The University of Illinois Press, 1949), p. 23.
3. C. Jackson Grayson, Jr., Decisions Under Uncertainty (Boston, Massachusetts: Harvard Business School, Division of Research, 1960), p. 70.
1. Glover, William L. and Lenz, John O., A Cost Growth Model for Weapon System Development Programs, Unpublished Master's Thesis, Wright-Patterson Air Force Base, Ohio: Air Force Institute of Technology, 1974.
2. Shannon, Claude E. and Weaver, Warren, The Mathematical Theory of Communication, Urbana, Illinois: The University of Illinois Press, 1949.
3. Grayson, C. Jackson, Jr., Decisions Under Uncertainty, Boston, Massachusetts: Harvard Business School, Division of Research, 1960.