Even the most meticulously planned and managed project can get into difficulty because it is an amalgamation of people, processes and plans in an ever-changing environment. In most situations it would be very valuable to have a method to assess a project's current state and continued viability. This paper offers a proven, non-quantitative technique for analyzing a project's health and potential risks.
The technique follows the precepts of project management outlined in the PMBOK® Guide, which can also be found in the key process areas forming the centerpiece of Software Engineering Institute Capability Maturity Model (CMM) Level 2 and 3 organizations. The method was initially developed for the State of Washington's Year 2000 Risk Assessment Program and used to assess state agency and university's Y2K information technology mitigation efforts. Several private consulting firms (Sterling Associates, Cotey Computer Services, CASE Associates, Management Technology Group, and CIBER) worked with the state's Department of Information Services to develop program and project risk assessment criteria and processes.
The technique was modified and extended within my organization and used to conduct risk assessments for both Y2K and non-Y2K projects and programs in the public and private sectors. The technique has been used successfully to assess both information technology and construction projects and is easily adaptable to most project environments.
Risk Assessment Process
Critical Success Factors
Central to the technique is a set of 10 Critical Success Factors (CSFs), which are used to evaluate the effectiveness (i.e., riskiness) of management, planning, resourcing, and other processes essential to a project's success. The 10 critical success factors are as follows:
1. The project is appropriately organized (Organization).
2. Project risks are identified and appropriately managed (Risk Management).
3. The project is appropriately planned (Planning).
4. Project milestones are being met on schedule (Milestones).
5. The project status is appropriately monitored and adequately controlled (Monitoring and Control).
6. The project scope is appropriately controlled (Scope Management).
7. The project is appropriately resourced (Resources).
8. Appropriate functional acceptance-testing processes and plans are in place (Functional Testing).
9. Appropriate capacity and performance acceptance-testing processes and plans are in place (Capacity and Performance Testing).
10. Appropriate and timely training is available (Training).
Depending on the project type and environment, these critical success factors may require modification or extension.
Each major critical success factor is further subdivided into sub-CSFs, which provide further details and criterion for risk evaluation. Each sub-CSF asks a specific question and has defined evaluation criteria to aid in assessing risk. If the evaluation criteria are missing or substantially incomplete, the sub-CSF is rated as HIGH risk. If the criteria is substantially complete but requires more work, it is rated as MEDIUM risk. Only if the criterion is completely met is it rated as LOW risk.
Critical Success Factor Evaluation
Findings for the sub-CSFs are documented with appropriate references to supporting interview notes and/or other documentation (i.e., “cataloged evidence”). When the sub-CSFs have been analyzed, an overall CSF assessment rating is set. To provide a conservative risk assessment, the following guidelines are used for each major critical success factor rating:
• If one-third or more of the sub-CSFs are HIGH risk, the CSF is rated as HIGH risk.
• If half or more of the sub-CSFs are LOW risks and there are no HIGH ratings, the CSF is rated as LOW risk.
• Any other combination is rated as MEDIUM risk.
As well as a rating for the major risk assessment category, a recommendation for action to reduce risk is created for each HIGH or MEDIUM risk sub-CSF.
A similar strategy is used to rate the overall project risk. If 4 or more Critical Success Factors are rated as HIGH risk, the overall rating is HIGH. If 6 or more ratings are LOW and there are no HIGH ratings, the overall rating is LOW. Any other combination is rated as MEDIUM risk. In the final risk assessment report presented to the client, recommendations for HIGH and MEDIUM risks are proposed as actions, which should be taken to reduce the risk assessment to LOW.
This technique requires little training and no project specific knowledge. A typical initial assessment can be completed in 20 to 70 hours of effort depending on the availability of necessary interviewees, the “state” of project documentation, and the required level of reference documentation and supporting evidence. Follow-up assessments require less time as “open” recommendations on HIGH and MEDIUM critical success factors become the focus of evaluation.
This technique concentrates on “correctness” of assessment rather than a numerical rating system. It is the findings and recommendations that are the core of the technique. That is, while it is important to identify the risk level and even boldly display it in color on presentations (i.e., High—Red, Medium—Yellow, and Low—Green), the paramount value is in the recommended corrective action to reduce risk and better ensure success.
Critical Success Factors
Following is a description of each of the Critical Success Factors and their sub-CSF evaluation questions. Exhibits 3 through 12 at the end of the article outline the evaluation criteria and acceptable documentation for verification of completeness for each CSF.
Organization (CSF # 1)
The Organization CSF assesses the breadth and depth of the project's organization to determine if it is structured to deal with both tactical and strategic issues. Projects without proper sponsorship and management oversight will incur many problems especially in issue and change request resolution. Also, lack of explicitly defined project roles and responsibilities can create confusion and decidedly impact decision-making processes. Three sub-CSFs are defined for organization:
• A project Steering Committee comprised of executive decision-makers is functioning?
• An Executive Sponsor from the business community has been designated?
• Project Management roles and responsibilities, with lines of authority and accountability, have been defined and agreed upon?
Risk Management (CSF # 2)
The Risk Management CSF provides an assessment of the risk identification, mitigation strategy and contingency planning for high probability and/or high impact risks. It, also, assesses the continuing validity of high impact assumptions. Proactive risk mitigation is key to a projects likelihood of success. Five sub-CSFs are defined for risk management:
• Have project risks been identified and categorized as to likelihood and impact?
• Are appropriate risk mitigation strategies in place with appropriate monitoring measures?
• For high probability or high impact risks, have contingency plans been developed in case the risk mitigation strategy fails?
• Is an ongoing risk identification, assessment and management process in place and operating effectively?
• Have project assumptions been verified and appropriate monitoring measures been put in place to ensure failed assumptions do not become risks?
Planning (CSF # 3)
The Planning CSF provides an assessment of the breadth and depth of project planning, scheduling and identification of external dependencies. While it may seem obvious to successful project management, an appropriately detailed and managed plan is an absolute requirement and should be closely scrutinized. There are six sub-CSFs to be examined for Planning:
• Are all appropriate tasks identified in the project plan?
• Are dependencies among tasks identified, including decision dependencies?
• Has a schedule been established and is it reasonable based on resources, productivity assumptions and dependencies?
• Is the plan clear and detailed enough to monitor progress?
• Is the project plan used to track progress and updated on a regular basis?
• Are external project dependencies identified in the plan?
Milestones (CSF # 4)
The Milestone CSF assesses the scheduled completion of interim and major project milestones and their impact on overall project completion. Successful milestone completion is a true measure of progress and a probable predictor of the future. Three sub-CSFs are defined for Milestones:
• Are interim project milestones being met so far?
• Are major project milestones being met so far?
• Is there sufficient time (with appropriate slack) to complete the project before the committed completion date?
Monitoring and Control (CSF # 5)
Assessment of Monitoring and Control examines the project status reporting process for task completion and budget. Only by appropriate monitoring and control can the project manager and sponsor expect to comprehend status, address project slippage and take corrective action. The Milestone CSF has for sub-CSFs:
• Does the project receive appropriate and timely executive and business sponsor attention?
• Are project status and activities being monitored and reported in enough detail and with enough frequency to ensure early detection of problems or schedule slippage?
• Is the project budget being appropriately tracked and reported?
• Are external project dependencies included in status reporting?
Scope Management (CSF # 6)
Scope Management assesses the manner in which issues, change requests and configuration management are implemented and followed. Ineffective scope management has been demonstrated time and again to be one of the key reasons that projects are over budget and late. The three sub-CSFs for defined for Scope Management:
• Are issues appropriately identified, escalated, and resolved in a timely manner?
• Are change requests effectively recognized, analyzed for impact, and approved prior to inclusion in the project scope?
• Are appropriate configuration management practices in place and being followed?
Exhibit 1. Risk Assessment Project Outline
Exhibit 2. Extract of Risk Assessment Working Papers
Resources (CSF # 7)
The Resources Critical Success Factor is designed to assess the capacity and skill set of the assigned project staff (development and maintenance). Project plans are based on the availability of required resources; if they are not available, the likelihood of failure is high unless treated as a risk and mitigated. There are five sub-CSFs associated with Resources:
• Is the level of effort estimated for each task at an appropriate activity level and is it reasonable?
• Are appropriate staff resources (skill set and quantity) available and assigned to complete project implementation?
• Are appropriate staff support resources (skill set and quantity) available and assigned to provided ongoing maintenance and enhancement?
• Are appropriate tools and computing capacity available and effectively utilized?
• Does the project have a sufficient budget to conduct required activities?
Functional Testing (CSF # 8)
The Functional Testing CSF provides an assessment of the functional capabilities of the system against current operational needs. Incomplete functional testing will be extremely detrimental to successful implementation; also, operational and environmental requirements can change during the life of a project. Five sub-CSFs are defined for Functional Testing:
• Do the currently contracted functional specifications match the current operational needs?
• Are the owning business users and management involved in establishing the functional acceptance testing scope and standards?
• Are functional acceptance test processes appropriate, and are results monitored and tracked?
• Are functional system interface test plans developed, followed and tracked?
• Is comprehensive end-to-end functional acceptance testing performed or planned including testing of all software, hardware and telecommunications components?
Exhibit 3. Organizational Critical Success Factor
Exhibit 4. Risk Management Critical Success Factor
Capacity and Performance Testing (CSF # 9)
The Capacity and Performance Testing CSF provides an assessment of the capacity and performance capabilities of the system against operational needs. Even if functionally perfect, if the delivered system cannot sustain real-world loads and expanding scalability requirements, implementation will be judged unsuccessful. Six sub-CSFs are associated with Capacity and Performance Testing:
• Do the currently contracted capacity and performance specifications match the current operational needs?
• Are the owning business users and management involved in establishing the capacity and performance acceptance testing scope and standards?
• Are capacity and performance acceptance test processes appropriate, and are results monitored and tracked?
• Are capacity and performance system interface test plans developed, followed and tracked?
• Is comprehensive end-to-end capacity and performance acceptance testing performed or planned including testing of all software, hardware and telecommunications components?
• Are infrastructure conditions (down to the computing hardware level) which may affect the application being considered, tested and resolved?
Training (CSF # 10)
The Training CSF provides an assessment of the training plans and materials for administrative, support and user staff. Historically, training has been one of the first deliverables to be sacrificed when a project encounters difficulties. However, history has also shown that without appropriate training, successful implementation is at best problematical. Training has four sub-CSFs:
• Appropriate training materials are available for administration and support staff
• Administration and support staff training is appropriately planned and scheduled
Exhibit 5. Planning Critical Success Factor
Exhibit 6. Milestones Critical Success Factor
• Appropriate training materials are available for system users
• System user training is appropriately planned and scheduled.
Executing a Project Risk Assessment
Whether performed internally or as an external validation and verification process, a project risk assessment should be conducted as objectively and professionally as possible. The purpose is not to condemn the project team or management but to identify and mitigate risk. It is to everyone's advantage to conclude a project successfully—on time, within budget and satisfying user (owner) requirements. Therefore, a clear statement by the assessment sponsor is required to ensure both cooperation and accurate information from the project team and owners.
A project risk assessment itself should be conducted as a mini-project. This implies following the same rigor and formality as with any project. A charter (perhaps a standard one) should be defined which identifies the assessment sponsor, assessment scope and objectives, participants with their roles and responsibilities, assessment plan and schedule, etc.
Exhibit 7. Monitoring and Control Critical Success Factor
Exhibit 8. Scope Management Critical Success Factor
Exhibit 9. Resources Critical Success Factor
Exhibit 10. Functional Testing Critical Success Factor
Project Schedule
As stated earlier, an initial project assessment of this type can be conducted in approximately 20 to 70 hours of effort spread over a one to four week period. Follow-up assessments can be conducted in a shorter period of time because a baseline is established and the focus is on open recommendations to mitigate risks discovered in the prior assessment. From our experience, a typical task list and schedule are presented in Exhibit 1.
Depending on the size of a project, even 20 hours of assessment time may seem unjustifiable. However, if a project is in trouble then even 70 hours to determine root causes and take corrective action may be easily justifiable. For projects that appear to be progressing successfully, there may me no reason to execute a formal assessment; nevertheless, a risk adverse project manager could use this risk assessment technique as a confidence supporting self-assessment checklist.
The variance in effort and elapsed time is dependent on several factors: the level of required documentation, organization of existing project documentation, and availability of project team and sponsor personnel for interviews. During the State of Washington Y2K risk assessments, the high end of effort and elapsed time estimates were experienced as the assessment results had to pass audit-type standards. It should be noted that interview notes are considered “weak” evidence; consequently, every effort should be made to obtain “hard” evidence as documentation to validate both the assessment results and the risk mitigation recommendations.
Project Deliverables
In addition to the risk assessment working papers (see Exhibit 2), there are a number of other supporting documents produced during the assessment process.
• Interview Log: list containing interview control number for reference in assessment working papers and interview notes, interview date, interviewee name and title, and interviewer name.
• Documentation Log: list containing document control number for reference in assessment working papers, brief description of the document content, reviewer name, and reviewed by date.
• Risk Assessment Summary: synopsis of the sub-CSF level risk assessment content summarizing the findings, assessment rating and recommendations for the 10 Critical Success Factors.
Exhibit 11. Capacity Testing Critical Success Factor
Exhibit 12. Training Critical Success Factor
Concluding Remarks
While possibly not an expected result, our clients’ reactions to this kind of risk assessment were usually quite positive. For the most part, project managers recognized that the assessment was objective and not a personal attack or condemnation. Not only did the assessments point out areas of risk; but also, by responding to the recommendations, the projects did reduce their risks during subsequent assessments and were much more likely to meet their projected completion targets. It should be noted that the State of Washington did not suffer any serious Y2K related system failures or interruption of vital services.
Project sponsors were uniformly positive. Although they did not like hearing about potential risks, they were appreciative of the effort because it allowed them to take remedial action (as a rule) before it was too late. In addition, the assessment process demonstrated to both project sponsors and the intended system users that they play a critical part in project success. This realization almost always led to increased user involvement with its positive impact on project progress.
As an added benefit, many assessment clients began to incorporate the content of the ten Critical Success Factors into their project management approach. It may be no surprise that this was usually at the request/demand of project sponsor management rather than the technical project organizations. While by no means a formal project management improvement process as defined by the SEI Capability Maturity Model, this project assessment technique can help move an organization in the right direction.