Leveraging expertise

Introduction

The PM CoP portal and four initial communities (Contract Management, Risk Management, Systems Engineering, and Total Ownership Cost) are being established to help the program manager, the program's executive team, and their industry partners to perform their jobs more effectively. The concept is to provide knowledge anytime and anywhere by linking acquisition community members and supporters, horizontally and vertically, to solve problems and learn. Knowledge reuse and creation are the cornerstone attributes aimed at increasing workforce proficiency and decision quality. Knowledge contributions, hereafter referred to as job aids, include acquisition course material; government and industry databases; training and online learning resources; best practices and lessons learned; examples and case studies; subject matter experts; web sites; etc. The PM CoP portal provides a web-enabled collaborative and sharing environment integrated into DoD acquisition processes.

The PM CoP risk management community, which currently has over 500 members including many the top government and industry risk professionals, provides just-in-time learning, collaboration and performance support tools to improve overall on-the-job risk management performance. The intent is for government and industry acquisition workforce personnel to access and use the site while performing risk management tasks or addressing risk management problems. Risk Community members contribute examples (actual risk management plans, risk identification forms, watch lists, etc.), templates, lessons learned, training materials, references and policy and guidance information that are then integrated with the learning content.

Community Based Approach to Supporting Risk Management Performance

The PM CoP team is attempting to leverage the expertise of government and industry risk professionals from all areas of the acquisition workforce. The idea behind this is that most of complex problems in any field have already been addressed by risk management practitioners. The challenge is to determine how to use and apply their experiences to aid others. The approach taken by PM CoP is to entice, excite, and engage key risk management professionals to identify the types of problems and concerns that exist, the types of tasks that need more support and the types of questions that most practitioners are concerned with.

Based on extensive community interaction data from both government and industry risk managers, the PM CoP Risk Management Focus area has worked with RM community members to develop a set of questions, tasks and problems that the RM community has stated that if addressed, would aid in significantly improving the overall acquisition workforce risk management performance. This list of questions, tasks and problems are online in the risk management community connection area. Through information collected in face-to-face risk community meetings among novices, practitioners and experts along with job aids provided by RM community members, new task guidance has been developed, all community developed risk questions have been answered, and new risk dilemmas and problems have been identified and addressed. These in total have provided acquisition workforce personnel better overall guidance and understanding for doing risk management.

The approach taken for building the online community site is an explicit merging of learning assets developed by instructional designers (then validated by community subject matter experts) and job aids developed and submitted by risk management practitioners. Job aids include completed documents (actual risk management plans and risk identification forms, for instance), templates for creating RM data (including RM plans and watch lists, for instance), lessons learned associated with all areas of the risk process, learning materials developed and used by community members, online discussions, case studies, useful risk tools, etc.

Questions

Over 100 key risk related questions identified by PM CoP RM community members have been answered in the following categories: Basic Risk Management; Risk Management Groundwork; Risk Management Responsibilities; Government Contractor Relationships; Schedule Risk; Risk and the Acquisition Life Cycle; Software Risk; Cost Risk and Technical Risk. For each set of questions, a recognized risk management subject matter expert reviewed and approved all content prior to its uploading for online dissemination.

Task Support

The PM CoP Community members provided a set of risk related tasks that they felt required more guidance for many to successfully accomplish. For each task, it was determined that the appropriate risk fundamentals information be incorporated directly into the task guidance. Additional planning and implementation information is also provided for each task. Job aids developed by RM community members are associated with the task to provide sufficient detail to allow practitioners to successfully complete the task. As an example, for the “Create a Risk Management Plan” task, there are over 50 pages of Justin-time learning content devoted to understanding and planning risk management plans. Examples of solid risk management plans, templates for creating RM plans, tools, and lessons learned for developing RM plans that were all contributed by RM community members are then associated with the understanding and planning content.

Community Based Risk Projects

One way to generate information on a key risk management subject is to galvanize community members to participate in a risk related community project. The PM CoP RM community is working with the PMI Risk SIG and the INCOSE Risk Management Working Group (RMWG) to in collecting and disseminating Risk ROI data in coordination with the PMI Risk SIG and the INCOSE RMWG. We anticipate this community based project will yield both formats for collecting future ROI data and actual case studies for showing real ROI savings based on RM.

Community Approach to Problem Solving

The PM CoP Risk Management team has been using face-to-face community meetings to develop guidance on problems facing the risk management community. So far, two key problems have been address by our community: “Choosing techniques for determining probability” and “How do you justify or sell the importance of risk management?” For each problem, a group of 15–20 risk management practitioners and experts came together to discuss and elaborate on these issues. Based on these meetings, content was written up and distributed for comments by the meeting participants and community core members. This process led to the creation of new guidance to these problems that is now being disseminated across the acquisition workforce. Due to the short length of this article, I can only cover this content in part. To see the full versions of these, along with the RM community members who participated in these discussions, go to http://www.pmcop.dau.mil.

Problem: Choosing Techniques for Determining Probability

Statement of the Problem

What techniques are available to aid in determining probability of risk events? How do you know which one to use given your circumstances? How do you know if you've implemented it correctly?

Background

This problem originated from Gerry Freisthler, the former Deputy Program Manager for the F22 program office. He mentioned that this problem had a big impact on cost and schedule and that it was always hard to come up with accurate probabilities. The issues involved centered on the selection of the appropriate probability technique for the circumstances. More importantly, it was difficult to know if they had done a good job on determining probability. They usually had to wait for the future to determine if it was done correctly. This issue was discussed and elaborated in the RM community meeting on 13 December 2001. This issue is problematic in potentially all areas of the risk management process model.

Inputs to Determining the Appropriate Probability Technique

The determination of the appropriate probability technique is only relevant after the appropriate actions have been taken. There were a number of actions that were detailed, including:

•   An open discussion of risk events is necessary

•   The quality of the risk events was is critical

•   Are people focusing on probability or the consequence, whether people were focusing on probability or the consequence

•   Key Performance Parameters (KPPs) must be in place to conduct accurate risk analysis.

•   Use of an integrated project schedule is critical for integrating RM into overall project management functions.

Qualitative or Quantitative Approach?

The determination of whether a qualitative or quantitative approach will be employed is the key decision for determining the appropriate Probability Technique. Qualitative risk analysis is a process of assessing the probability/likelihood and consequence/impact of identified risks using ordinal or cardinal scales. Quantitative risk analysis is a risk analysis technique that uses a single number to portray the risk associated with a particular event. There are a number of factors that aid in determining which approach should be taken. In many cases, a hybrid approach that involves both a mix of qualitative and quantitative risk assessment may be the best solution. Factors involved in making this decision include:

•   Team risk skill levels. The probability technique is based on the skill levels of the individuals on the team. Additionally, good quantitative risk analysis requires more skill on the part of the project/program.

•   Historical data availability. Historical data from past projects is very useful if it has similar technology, scope, duration and cost. If you have accurate historical data, a quantitative approach for determining probability may be preferred.

•   Is this an independent risk assessment or your own project? If you are conducting an independent risk assessment, the probability technique chosen will often be associated with a larger process used for conducting independent risk assessment. However, due to time constraints, most independent risk assessments are qualitative in nature.

•   Individual program/project or portfolio? Portfolio management, in which a set of programs is being rolled up and looked at in total, has different risk management needs than individual program risk management. Portfolio risk management is at more of a strategic level, and may be attempting to quantify the probabilities from across the set of programs in the portfolio. This implies a different level of analysis and precision.

Exhibit 1. Probability Techniques Identified by Community Members

•   Organizational culture match: The method you choose should coincide with the culture of the organization. If, for instance, you are in a very “numbers” oriented culture, a quantitatively based, number approach may be better.

Types of Probability Techniques Identified

Exhibit 1 provides a list of the probability techniques identified at the community meeting. While not exhaustive, the community members felt that the majority of the key options were listed below.

Lessons Learned

To work this problem, there are a number of pertinent lessons learned which may apply to your situation:

•   Good risk assessment involves good data collection. This is based on the maturity of the risk management process and the skills of the people involved. Training your people is essential. Also, interviews with stakeholders, SMEs and participants are essential to good data.

•   Take qualitative data and then quantify it. In combining the two approaches, Qualitative risk analysis is generally performed first and quantitative analysis can, then, be used together with the qualitative analysis phase or used separately from it.

•   How do I convince the boss to use this risk assessment content? Gather as much data as possible (technical, cost, schedule, political, etc.) and provide a “business case” to senior management. If you have data, preferably quantitative but even qualitative, a strong argument, based on technical and nontechnical data is a big hammer.

•   Use of expert opinion in material. The use of expert opinions should be important enough to expend project funds on to elicit outside subject matter experts and risk experts.

•   Communication is the key to good RM. Good risk management relies on effective communication. Feedback is necessary and critical to good communication.

•   Organization structured for effective RM. The PM will need to organize the program office and establish interaction with external organizations in order to manage risk. Risk management is an integral part of program management and not an additional or separate function to perform.

Pitfalls

There are many pitfalls associated with this problem. The list below summarizes the main items to be aware of:

•   Only focusing on the top risk events. It may be that the majority of the cost/schedule/performance impacts are in the lower risk events. Even though the focus is on the “top 10,” the totality of the lower risks may very well have the potential to kill the program.

•   Paralysis by analysis. At some point the dataset must be seen as complete.

•   Sacrificing Accuracy for Precision. There is a danger in employing quantitative risk analysis techniques in which the data doesn't lend itself to becoming extremely precise. Quantitative risk analysis techniques can make the data seemingly appear extremely precise, yet if the data doesn't support this, the output will be inaccurate, yet precise.

•   Mathematical operations should not be performed on scores derived from uncalibrated ordinal scales. One way to avoid this situation is to simply show each risk event's probability/likelihood and consequence/impact separately with no attempt to mathematically combine them.

Conclusion: How Do You Know if You Did This Correctly?

The final question Gerry asked was how did they know if they did the probability determination correctly? The community has come up with the following related responses:

•   Gut feel. You should look at the results of the risk assessment and see if they match gut feel. If this doesn't match your gut feel, chances are something may be wrong.

•   When to stop. Generally, one keeps going until the remaining risks all fall into the “acceptable” range.

•   Firefighting decreases significantly. The number of project” fires/problems” should decrease over time.

•   Process meets performance objectives: The risk management review board should establish process performance metrics and objectives. The process metrics must be reviewed periodically to evaluate the process performance.

Problem: How Do You Justify or Sell the Importance of Risk Management?

Statement of the Problem

Many community members have described problems in getting programs to incorporate Risk Management into their program. Throughout the acquisition workforce, selling or justifying the importance of risk management is problematic. Although risk management is required by policy, too many programs both large and small have not integrated risk management into their program management processes. Implications to this include cost overruns, schedule slippage and inability to meet KPPs.

Background

In many instances programs will ignore any formal RM process, give lip service to RM, or simply not fund it. In attempting to get both government and industry program personnel to support and integrate risk management process into program office operations, risk proponents have continually run into roadblocks. When posed with this situation, risk proponents are then put in the position of attempting to justify or sell the importance of RM. In this situation, how does the risk proponent go about convincing leadership to expend resources on establishing a risk management program or convince the rank and file required to actually do the risk management? The goal of working this problem is to provide resources for risk proponents to use in justifying or selling the importance of RM to their program. This issue was discussed and elaborated in the RM community meeting on 28 February 2002.

Context: When Does This Problem Occur?

This problem occurs in which one or more of the following situations exist:

•   Inadequate skill sets. Programs or projects which do not have either the familiarity or skill sets associated with risk management.

•   No leadership support. Programs in which the leadership, (government, contractor, or both), does not buy in to enforcing a formalized risk management process. If leadership is not convinced, chances are this problem is insurmountable.

•   Non-trusting environments. Situations that foster a nontrusting or guarded environment in which risk events cannot be openly discussed.

•   Programs with unrealistic budgets or schedules. Risk management processes will not be enacted when insufficient time or dollars are allocated. In programs that have undergone massive funding cuts, risk management is often looked at as an add-on, and not part of the core of program management.

Problems Encountered in Justifying or Selling RM

The following problems in justifying RM were encountered by our community members:

•   Culture. Some organizational cultures do not foster good risk management practices. Often these cultures are referred to as “blame oriented” or “shooting the messenger” cultures. In this situation, the risk management process, along with many other communication and program management aspects suffer greatly.

•   “We have a Risk Management Plan, therefore we have no risks.” Some programs have had the belief that if they have a plan for mitigating the risks, then the risks are already mitigated, and are no longer considered. This ignores the notion that risk management is not a one-time event, but is instead an ongoing process

•   Inability to invest risk management dollars to alleviate the impact of high risks. In many cases, the dollars are not provided to effectively mitigate the risk events. The risks are then effectively “assumed.”

•   Nothing catastrophic has occurred. Until a program encounters a major catastrophe, the risk management process may not get the attention it deserves.

•   Not following the letter of the law. Lip service to risk management is provided but real actions to identify and mitigate risks are not enacted.

•   Spending money on fancy tools is not always appropriate. In any implementation, the tool rarely makes the difference between success and failure.

•   Contractor is not graded in the RM processes; they are graded on the final product. Sensitivity on the contractor side for doing RM is often minimal. If RM can't “prove” any successes, the outcome is often that the contractor will try to minimize expenses.

•   Lots of small risks eventually can amount to cost or schedule impacts. Unfortunately, a large number of small, minimal impact risks can create problems that affect program schedule and cost.

Lessons Learned to Justify Funding RM in a Program?

The following approaches have been taken by community members to justify RM funding:

•   Quantitative negative numbers convince people to do RM. To be effective, every attempt to back up your cost numbers must be taken.

•   Take advantage of catastrophes. Catastrophes can be described as significant events that cause major cost overruns, schedule delays, or worse. When catastrophes occur this is usually a great time to advocate a sound risk management process to address future unknowns.

•   “Reduced net costs at the end of the project” is achieved through identifying and mitigating risks. The cost of a sound risk management process should be compared to the costs of having additional money put into the project for “fighting fires.”

•   Ask, “Have we accounted for the problems that have occurred in the past on similar programs?” This train of thought should lead to a forward thinking approach to managing risk.

•   Motivating briefing to senior leadership. Unless you get senior leadership support for your RM process, you will have an uphill fight.

•   It's about risk training, not selling. Engage people in motivating risk training workshops that force them to think of potential risks currently facing their program that they had not previously thought of nor managed. This training time will now be seen as valuable work versus wasted independent training.

•   Strict limits on dollars and time. If you are in a situation where “If we don't do the job right, we're not going to get there,” the rationale for engaging in a focused risk management process increases dramatically.

•   “What's in it for me?” Syndrome: If both the government and contractor representatives, specifically the IPT members are involved in the risk management process there is usually better buy-in than if only a few individuals set up and manage the process.

•   “Incentivize” the contractor. Contractors will generally do whatever it takes to get all the money. Rewarding good risk management should be written into the contract.

•   Employ independent risk assessment reports. A third party analysis of a program or the program's contractor will often identify risk events that those close to the problem will not recognize, and will also highlight RM in the eyes of senior leadership.

Strategies to Prevent This Problem from Occurring

The following strategies were determined to be effective to either stabilize an existing RM program or to prevent this problem from occurring:

•   Create a risk reserve account. The creation of a risk reserve account is a great way of solidifying a risk management approach within a program, and to effectively mitigate current and future key risks.

•   Risk management needs to be incorporated into the entire program if it is to be successful. Unless risk management is institutionalized across the board, the potential for problems arising is higher than necessary. Trust is required.

•   Simple is better. A simple risk management process is more likely to be adopted and adhered to.

•   Integrate continuous improvement methodologies. In situations where there are RM activities that are repeated, the significant risk management savings occur in the third, fourth and later times that the task is enacted.

•   Spend resources on risk training. Continuous risk training for both government and contractors will aid in keeping interest and support for your risk management process. Please utilize the PMCoP Risk Management Fundamentals to augment your risk training. This is a way of providing free training to your entire supply chain.

•   Require CMMI certification. The government can require CMMI level 2 or 3 assessment of vendors prior to establishing contract and require the CMMI level be maintained during the life of the contract. This ensures that the contractors have to at least have a documented and assessed risk management process prior to award.

Future Direction

The PM CoP Risk Management team will continue to provide support for the questions, tasks and problems identified by our community members as requiring additional support. It is anticipated that at least once every other month, a new problem will be detailed to provide additional guidance. All identified questions and tasks should be online by the time this paper is published. This effort is only successful to the extent that the risk management practitioners participate and interaction in the RM community. It is their expertise and knowledge that is leveraged.