- How do we evaluate and select the most valuable projects?
- What information and measurements can be collected to help evaluate projects?
These types of questions are best answered by using leading practices during the project ideation and initiation phase. This paper will focus on each step of the project ideation / initiation process, beginning from the filtering of the opportunity through the gathering of requirements, the development of estimates, and the creation of the statement of work. Since inherent in a solid project initiation process is the capturing of key metrics, we will discuss some key metrics as well as additional project management metrics that, when captured throughout the life cycle of the project, form the basis for sound process improvements.
This paper is organized in two distinct sections complementary of each other. The first section, explores a set of processes that are designed to help organizations make more informed decisions when selecting a project. The journey starts by laying the foundation blocks where information is stored, managed and made available to all interested players in the decision making process. A deeper dive into each of these blocks will bring focus to what is required as input and output that ensure a tight integration between them.
The second section, defines a measurement program that encompasses the full project life cycle to ensure the effectiveness of the project initiation phase. The Goal Question Measure (GQM) and Goal Question Indicator Measure (GQIM) approaches are briefly defined to set the stage for a discussion of the different types of measures and some examples of each. The implementation of the measurement program is outlined to close the loop.
The conclusion of the paper presents options of tactics that if implemented correctly, will help address three common questions decision makers ask about projects:
- How do we evaluate and select the most valuable projects?
- How do we collect information and measurements to help evaluate projects?
- How do we increase speed to market?
The Webster's Desk Dictionary defines standard as: “1. something accepted as a basis of comparison. 2. a rule used as a basis for judgment. 3. the authorized exemplar of a unit of weight or measure...” Simply put, a standard is a set of rules and guidelines that provide a common framework for communication and understanding.
The foundation for project evaluation is a set of guidelines designed to help the decision makers do what they do best: make decisions on what ideas should become projects to bring the most value into the organization. The business environment and landscape has changed dramatically in the past 10 years with the introduction and advances in technology. Businesses and consumers are more sophisticated and technically savvy than prior generations. The appetite for instant gratification has also increased and faster and bigger returns are expected. However, the supply of resources both human and financial are forever more scarce. Therefore requests for new Information Technology services and investments require more scrutiny and justification.
To address these concerns and challenges, successful organizations are establishing and enforcing standards for evaluating potential projects and requiring consistent information such as entry criteria, justification (i.e. Return on Investment – ROI), estimation processes, and centralized repository of these data for easy access and analysis.
The value added of the standard foundation and the evaluation process is the increased client satisfaction. Early filtering of ideas to invest only in the projects that add value, and clearly defining features and functions will improve project definition and influence the quality of results.
The increased quality of output will directly lead to increased cost effectiveness gained only by executing a consistent and repeatable project definition process. The project definition process represents the initiation phase of the Software Development Life Cycle (SDLC) that seeks to understand the business needs and recognize its strategic or tactical direction; identify, evaluate, and propose solutions that satisfy the defined business needs; and gain stakeholder commitment to scope, investment and timeframe.
During the Filter Opportunity Step 1 of the process, the request submitted by the business client is evaluated and the reviewer validates that the required information is included. The organization responsible for managing the business requests conducts an internal capacity check to ensure that the request gets slotted, prioritized and a manager gets assign to lead the process.
The responsibility of the assigned manager in the filter phase is to ensure that the business needs are met and communicate any immediate concerns raised by the request. A client interview technique is used to obtain more clarity around the request and ask more detail questions about how the request fits into the major business initiatives; get a sense for the impact of the request unto other business units' processes or initiatives. If the request crosses boundaries of business function groups, the manager's responsibility is to help the requesting business unit attain alignment between all groups affected.
At this stage of the process, the preliminary scope statement is drafted documenting the parameters or boundaries of the request. The assigned manager begins to develop the plan to execute and put in place all the required activities needed to accomplish the ultimate goal of creating the Statement of Work (SOW). A critical activity is the identification of areas involved in the request. By doing this, the manager effectively is building the core team that will be working together through the duration of the process and submits the staffing request to the appropriate technology and business groups. The basic roles of the core team are as follows:
- Information System Architect
- Application System Architect
- Data and Database Architect
- Business Process Analyst
- Estimation and Metrics Subject Matter Expert (SME)
A kick off meeting is scheduled with all stakeholders including the business sponsor submitting the request. The objective of the kick off meeting is to communicate the scope of the request, desired schedules and expected deliveries. Alignment and commitment from all areas involved is obtained during the meeting. Step 2 of the process gets into exploring options available with the established core team. Many techniques are used during this process: brain storming, consensus, directive management just to name a few. The decision of implementing one methodology versus another is up to the person leading the process.
The architects conduct an impact analysis of the request and are accountable for providing architecture direction via a formal conceptual design of all viable options. The estimators work based on the conceptual design provided by the architects in collaboration with the application subject matter experts. All assumptions, risks, mitigation strategies and issues are documented along the way of the process and that will become part of the proposal to be presented to the business client. The following content is the base of the proposal:
- Business Need
- Current State – Business Process and supporting applications
- Objectives and Preliminary Scope
- Assumptions, Risk, Issues
- Preliminary Requirements – Functions or Features
- Options and Estimates
In Step 3 the completed proposal is presented and reviewed with the business sponsor. Further analysis might provide an opportunity to iterate the proposal based on a refined scope, features & functions, estimates or any new information around assumptions and risks. The process is designed to be flexible in nature to capture accurate information that will help the business sponsor make a more educated decision to move forward.
The final Step 4 is designed to obtain final commitment from the business and technology organizations to do the work outlined in the proposal. Key activities in developing the SOW include the solution agreement, preliminary project resources identification, Software Development Life Cycle (SDLC) timeline at a phase level and key stakeholder's sign off.
The Full Project Life Cycle (FPLC) methodology will govern the project once it has gone through the appropriate governance process and a Project Manager is assigned to lead the project. But this is not the end of the process, we still need to complement the FPLC with a measurement program designed to ensure the quality of the project.
A good place to start is by building a solid foundation that is based on the Capability Maturity Model Integrated (CMMI®) which is a process improvement approach put forth by the Software Engineering Institute (SEI) that provides organizations with the essential elements of effective processes.
So, what does CMMI® say?
- SG 1: Align Measurement and Analysis Activities
- - SP 1.1-1: Establish Measurement Objectives
- - SP 1.2-1: Specify Measures
- - SP 1.3-1: Specify Data Collection and Storage Procedures
- - SP 1.4-1: Specify Analysis Procedures
- SG 2: Provide Measurement Results
- - SP 2.1-1: Collect Measurement Data
- - SP 2.2-1: Analyze Measurement Data
- - SP 2.3-1: Store Data and Results
- - SP 2.4-1: Communicate Results (Chrissis, Konrad, Shrum, CMMI® 2003)
The Goal Question Measure (GQM) and the Goal Question Indicator Measure (GQIM) are two approaches that have long been industry standards in guiding organizations in choosing appropriate metrics. The GQIM is similar to GQM but it also defines a representation of metric data that provides insight into software project or project improvement activity.
The process begins by developing the objectives of the Measurement Program. Those objectives are merged with the goals of the organization to identify the organizational goals that will be the focus of the program. An example of an organizational goal is that of improving the Timeliness of Change Request Processing.
The next step then is to begin to ask questions about each of the goals; an example of a question about the change request processing goal might be: What is the current change request processing speed? The answers to the questions developed are used to identify the measures. In the example above, some answers to the question might be Average Cycle Time or % of Cases above the Acceptable Change Request turnaround time. The measures selected should be only those that can be effectively reported on.
The last step is to identify which of those measures are base metrics. That is, those metrics that can be collected directly, such as cycle time for each project, and which of those measures are derived metrics, which are calculated using the base metrics that are collected. An example of a derived metric might be Average Cycle Time.
There are two types of measures: Strategic and Tactical. Strategic measures are those measures that help us to improve the organization. Questions that may be answered by these measures are:
- How is our productivity?
- Are we finding defects in the phase they are introduced?
- Are divisions meeting their quality goals?
- Are our inspections effective?
- Is scope and risks being managed?
Some examples of Strategic Measures are:
- Requirements, effort impact of requirements changes
- Quality, defect containment
- Process Effectiveness, how effective is the peer review process?
Looking at the overall impact of requirements changes or the effectiveness in identifying defects in the phase that they are introduced, may help us understand how effective the project initiation processes are.
Tactical measures are those measures that help track and control projects. Questions that may be answered by these measures are:
- Are we finding enough defects?
- Are we efficient?
- Are we following the processes?
- Are we going to make the dates?
- How accurate are the estimates?
- Are we delivering according to the plan?
Some examples of Tactical Measures are:
- Requirements, number of requirements changes
- Estimation Accuracy, last baseline to project actuals
- Process Effectiveness, average number of deviations per peer review
Looking at the number of requirements changes or estimation accuracy can help us to tighten up the project initiation processes.
Now that we have the measures defined, we need to implement the measurement program. First, there is a need for a measurement vision that will define the program, and it should be developed using key business goals in collaboration with the different stakeholders. It is very important that the vision encompass the full software development life cycle (SLDC) into account when developing the measures.
The next step in the building of the measurement program is to develop the data collection approach, processes and tools. This is what actually makes the measurement program a reality. We need to clearly communicate the measurement program to all stakeholders. This provides visibility into the measurement program and clear documentation of value.
A key factor for the success of the program is to automate as much as possible the collection and report of the data. A central repository will facilitate the analysis and sharing of data. Once the measurement program has been constructed and is functioning as designed, it is important to continue to perform the care and feeding of the program. Demonstrate the results by communicating the successes of the program itself and the successes of projects and process improvement efforts.
We must also do some introspection to regularly provide improvements to the program. On a regular basis, reevaluate the goals that are in place and verify whether the measures are still effective. Determine if the measures are being used for what they were intended, since measures enable process improvement, and reevaluate the process improvement initiative to ensure that it is focusing on improving the right processes.
The evaluation and selection of the most valuable projects is enhanced by gathering standard information that will help in the analysis if the data. For the process to be effective and unbiased, standard justification criteria must be applied to all projects ensuring that standard processes are followed and executed.
To support the decisions made, an organization must develop and collect measures based on goals that are aligned with the organization's key initiatives. Constant and timely communication of the measurement plan is also a must to sustain support by demonstrating the results of the measurement plan. Never forget to periodically reevaluate the measurement program to make adjustments when and wherever necessary based on the natural progression of the environment.
When an organization maintains and references historical data, jump starts the project initiation phase by using proven templates and delivers clear requirements, functions and features; it has effectively reduced the time it takes to deliver a solution, therefore, gaining a competitive advantage.