Measuring the success of project management and the capability maturity model using the parking lot metric
This document will demonstrate how the Lansing Solution Center utilizes Project Management along with the Software Engineering Institute’s Capability Maturity Model (CMM) to successfully complete software development projects on time and within budget.
The Parking Lot Metric
I conducted a Parking Lot Metric, a survey, of the number of cars in the parking lot after hours at the Solution Center. When I transferred to the Solution Center in February 1996, we had two large projects that were in serious trouble. People were working 12 to 14 hour days, six to seven days a week. The parking lot had anywhere from 50 to 100 cars late in the evening and on weekends. At that time the Solution Center had around 200 employees. The two large projects consumed half of our resources.
Today, the Solution Center employs around 350 people. My survey has less than 10 cars in the parking lot after 7:00 p.m. Many times there are no cars in the parking lot. This is especially true on weekends. What happened between February 1996 and November 1999?
I realize my survey was not scientific; however, the amount of long excessive project hours has decreased dramatically. The reason for this reduction is better Project Management capability and local processes. The vehicle that was used to increase our Project Management and process capabilities is the Software Engineering Institute’s Capability Maturity Model (CMM). The CMM ties in with the PMBOK® Guide for software development projects.
We reached CMM Level 2 in June of 1997 and reached CMM Level 3 in November 1999. The motivating factor behind our drive for the CMM Levels 2 and 3 was client requirements and competition from our competitors. We did take Project Management and process improvements to heart. Each and every employee understands the benefits and no one wants to return to the “old days” of marginal Project Management.
Overview of the CMM
The Capability Maturity Model is a business model that companies can use to help improve their software development capabilities. The CMM is a model, not the CM Law. Each company should apply the model so that it fits their business. I have worked with some project teams that follow the CMM down to every detail. This is not the intended use of the CMM.
The CMM is broken down into five maturity levels. The levels are:
Level 1 is the default level, so every company starts out at this level. Level 2 focuses on the software project’s concerns related to establishing the basic Project Management controls. Level 3 addresses both project and organizational issues. Level 4 establishes a quantitative understanding of both the software process and the software work products being built. Level 5 covers the issues that both the organization and the projects must address to implement continuous and measurable software process improvement.
Key Process Areas (KPAs)
Each maturity level other than Level 1 has key process areas (KPAs). Each KPA has a set of goals and has activities that help meet those goals. The CMM Level 2 is primarily Project Management. Level 2 has six KPAs. The KPAs that have had the greatest benefit for the Solution Center were Requirements Management, Software Project Planning, Project Tracking and Oversight, and Software Quality Assurance (SQA).
Requirements Management KPA
Requirements Management KPA of the Capability Maturity Model is how we define and maintain the scope of the project. It covers Scope Management in the PMBOK® Guide. It is a very detailed process that determines what is included in the scope of the project. It also helps us to keep the project within scope.
The Requirement Management KPA has two goals. The goals are:
1. System requirements allocated to software are controlled to establish a baseline for software engineering and management use.
2. Software plans, products, and activities are kept consistent with the system requirements allocated to software (SEI, 1993).
The first goal requires us to gather and formally document requirements and baseline those requirements. This means that we have to use a change control process to change those requirements. The second goal requires the project team to expend effort only if that effort help fulfills the requirements defined. That then means no “scope creep.”
The Requirements Management KPA forces us to go through a change control process when a change does come along. We research the change and determine the impact to the schedule, budget, and staffing plan. We then get client agreement to implement the changes to the schedule, budget, and staffing plan before we proceed. The client may agree to withdraw the change or to hold the changes until a later release. When that happens, the original schedule, budget, and staffing plan remains intact.
Before the Requirements Management process
One of the large projects in 1996 did not have any defined scope. The client kept taking advantage of us and insisted that changes and enhancements were within scope. Since we didn’t have any signed requirement or scope documents, we could not argue with the client. They received a lot of free enhancements at the cost of our team members. The team members spent a lot of overtime trying to include the new functionality and still meet the original deadlines. The client who received the “free” enhancements was not happy because we missed the deadline.
In the “pre-Level 2” days we would gather the requirements and develop the project plan which included the schedule, resource, budget, risks, etc. We would present the project plan to the client and they would tell us that the finish date was unacceptable. They almost always wanted the system in half the time we could provide it. Even if the Project Manager was strong enough to say no to the customer, our management would tell us to try harder and get the project in sooner. The team members would put in a lot of extra hours and try to get the project in sooner. As it turns out, it was rare that the project team finished by the artificial finish date. Most of the time, the project team would finish the project within the original scheduled finish date. The project team was set up for failure from the start. No one was happy; the client was unhappy because the project came in “late,” management was not happy because of the missed date, and the project team was unhappy because they spent a lot of extra time on the project, only to miss the artificial deadline.
After the Requirements Management Process
After reaching Level II, we gather the requirements and create the project plan just like we have always done. However, when the client starts telling us that the finish date is too far out, we start talking about the options we have to meet the new date. Sometimes we can add resources, which would increase the budget. A lot of times increasing resources will not shorten the project schedule much. The reason for that is when we develop the project plan, we try to optimize the resources. We then negotiate what functionality to cut in order to meet the new date. We can add the deleted functionality in later releases of the software. We let the client prioritize the functionalities they want and we plan for adding the functionalities in releases. This worked well for projects where the client has a definite hard deadline, like Y2K.
We still have clients trying to talk our management into moving the finish date earlier with all of the original functionality. Our management’s response is that we have to follow the processes or we would not be performing at a Level II capability. Morale of the project team really increases when management backs up the Project Manager and does not let the client push the artificial deadline. The client is not very happy at this point, however, they usually agree to one of the options that we provide them. In the long run, most clients become believers in the Project Management and Level II processes. The project teams deliver on time and within budget a greater percent of the time. The client knows exactly what he will receive and when he will receive it.
The Requirement Management KPA is one of the largest reasons we avoid the “death marches” that we were famous for in the past. We are getting better at controlling the scope creep that happened in the past because we are getting the requirements up front and creating realistic estimates. We then use change control to manage the changes that always creep up during the project. The project team members spends a reasonable amount of effort on the project and the quality of the software product increases.
Software Project Planning KPA
Software Project Planning is the next KPA in the Capability Maturity Model. The goals of the Software Project Planning KPA are:
1. Software estimates are documented for use in planning and tracking the software project.
2. Software project activities and commitments are planned and documented
3. Affected groups and individuals agree to their commitments related to the software project (SEI, 1993).
The Software Project Planning KPA covers several of the PMBOK® Guide Project Management Knowledge Areas. It covers the Project Scope Management, Project Time Management, Project Cost Management, Project Quality Management, Project Human Resource Management, Project Communications Management, and Project Risk Management. As you can guess, this is where project planning occurs. Our Project Management process calls this the startup and planning stages.
Estimates are generated from past project metrics and from past experiences of the team members involved, in addition to using an industry standard software-estimating tool. The project teams document how they estimate along with all of the assumptions so they can validate them during project execution. The estimates and assumptions are documented so if there is a problem during project execution, they can determine what estimate and assumption were incorrect so that they can replan to correct the schedule. It’s better to make the correction early in the project so expectations of the stakeholders can be set. By the way, the project teams go through a change control process when the schedule is changed.
The project team uses all three estimating techniques, past project metrics, experience of the team members, and the software-estimating tool, to get an accurate estimate. Obviously the three techniques will come up with different estimates. The project team will reconcile the differences between the estimates. This generally is a good check and balance. In one of our Web Development projects, the project team came up with an estimate of 10 resources for eight months. The estimating tool that we used calculated an estimate of three resources for three months. The project team had a hard time reconciling the difference in the estimates until they discovered a productivity tool that they were not using. Once they factored in the new tool, their estimate was relatively close to the estimate generated by the estimation tool.
The second goal of this KPA is software project activities and commitments are planned and documented. This goal is the standard Project Management activities that every Project Manager should perform. The first activity the CMM requires is “the project team participates on the project proposal team” (SEI, 1993). This allows the proposal to be more accurate and realistic. It also prevents sales from making unrealistic promises to the clients just to get the sale. Our company has turned down Requests for Proposals because of unrealistic requirements by the client. In the past, the sales force would bid on the proposal and the development team was stuck trying to implement it. It’s better to turn away unprofitable business and focus on the profitable projects. The secondary benefits of having the project team participate in writing the proposal is to get buy in. It’s easy to not be serious about deadlines when someone else makes the commitment, however most people will do everything they can to deliver on time and within budget when they make the promise.
The second activity of the Software Project Planning KPA that support the goals is “the project planning is initiated in the early stages of the project” (SEI, 1993). This is common sense; however, I know of a lot of projects where the programmers already started their tasks and are generating outputs before the project plan is created.
The third activity is “software project commitments made to individuals and groups external to the organization are reviewed with senior management according to a documented procedure” (SEI, 1993). This activity ensures that senior management is committed to the project and will support the project team throughout the entire project. The last thing the Project Manager wants is to make a commitment without the knowledge of management. If something goes wrong the Project Manager is on his own. It’s better to have management backing the Project Manager when things go wrong.
The next activity is “the software risks associated with the cost, resource, schedule, and technical aspects of the project are identified, assessed, and documented” (SEI, 1993). My site was not very good at managing risks before we reached the Level 2 maturity. One example is that we had a project that had new technology that we didn’t have a lot of experience with. We trained our programmers and were gaining experience with the new technology. The project was set back a couple of months because of the loss of a key resource. The Project Manager told me later that he was afraid that something like that would happen. I asked him why he didn’t plan for it and he told me that he had too many other fires to put out and never got around to planning for the risk.
Now we are much better at managing the risk. If we had another project with a very critical resource, we would define the risk and create a risk management plan, which would eliminate, mitigate, or accept the risk. If the critical employee left the company, the project would be in much better shape. The project would still be affected by the loss, however the consequence of that loss would be much less.
There are many other activities for the Software Project Planning KPA that I will not cover because of space limitations. Each and every activity in this KPA reinforces the Project Management activities defined in the PMBOK® Guide.
Software Project Tracking and Oversight KPA
The Software Project Tracking and Oversight KPA covers the execution stage of our Project Management Process. I remember creating a lot of project plans and putting them on the shelf when the project really gets going. I always meant to use the project plan that I created, however, I was immediately put in a fire-fighting mode and I never got back to the project plan. At that time, I did not have the discipline to do what I needed to do. Because I didn’t have the discipline, the project suffered. This KPA encourages us to have the discipline to track to the project plan. In addition to tracking to the plan, this KPA requires us to make adjustments to the plan when the project starts deviating from the project plan. That’s why tracking is so important to running a project. If there is no project tracking, how can the Project Manager make any necessary corrections?
This KPA has the following goals:
1. Actual results and performances are tracked against the software plans.
2. Corrective actions are taken and managed to closure when actual results and performance deviate significantly from the software plans.
3. Changes to software commitments are agreed to by the affected groups and individuals (SEI, 1993).
The first activity that supports the Software Project Tracking and Oversight KPA is “a documented software development plan is used for tracking the software activities and communicating status” (SEI, 1993). This activity should be obvious to good Project Managers. The Project Manager must have a documented project plan before he can track to the plan. If the plan is not documented or not complete, it is very hard to manage that project.
In addition to having a documented project plan, that project plan must be baselined. I cannot believe how many Project Managers do not baseline their project plan in a timely manner. I hear a lot of excuses for this; however the most common excuse is that the client has not signed off on the project plan. The Project Manager tells me that they have a good and trustful relationship with the client and they are proceeding with the project without having the signoff. Some of the projects I encountered are three quarters of the way through the project before the client signs off on the project plan. Proceeding this far without signoff is very risky. A lot can happen to the scope of the project and how can the Project Manager enforce scope and change management without a baselined project plan. My recommendation to those Project Managers is to baseline the project plan before starting the execution stage of the project assuming the client is trustworthy (there’s always a potential risk for proceeding without a signed off project plan). From this point forward, any scope changes go through a change control process including any changes required when the client finally signs off on the project plan. There may be some tough negotiations at that time, but at least the project team is in better shape to protect its interests.
The second activity for this KPA is “the project’s software development plan is revised according to a documented procedure” (SEI, 1993). In other words, the project plan is revised according to a change control process. The change control process calls for establishing a change control board. This board approves the change control, puts the change control on hold for a later release, or rejects the change control. The board is made up of at least the Project Manager and one of the key client stakeholders. There may be more people assigned to the change control board depending on the size and structure of the project. This change control board does not have to be formal. Most of the time it requires the Project Manager and the client to get together and approve or reject the change control request.
The project team must be disciplined enough to work only on the tasks authorized by the project schedule. The best change control board in the world will be useless if the project team members make unauthorized changes anyway. This takes real discipline. The team members want to make the client happy and are usually willing to do a little extra to make this happen. We always wanted to provide outstanding service to the client. However, we found out the hard way that providing outstanding customer service is tough when we do not follow the project plan.
The Project Manager must make sure that there is time allocated in the schedule to analyze the change control requests. When a change control request is submitted, someone on the project team must perform an impact analysis of the change and provide an estimate on changes to the effort, schedule, budget and/or resources. This takes time away from the tasks that the team member is already performing. If the change request is a large one that will take a lot of time, we will provide an estimate of the estimate. Sometimes the creation of the estimate will take several weeks. If the change control board agrees to the estimate of the estimate, the schedule is changed to allow time for the team member to create the estimate. Don’t forget the change control board approves or rejects all change control requests.
Another activity for the Software Project Tracking and Oversight KPA is “approved changes to commitments that affect the software project are communicated to the members of the software engineering group and other software-related groups” (SEI, 1993). This activity should be common sense. If a change is authorized, then the appropriate team members need to know about it so they can include the change in their work. In addition, rejected change requests should also be communicated to the project team. This is because most of the time, team members are aware of requested changes and may spend some time on the impact analysis. If the team member knows the change request is rejected, then they know they don’t have to worry about it and saves time. This activity is about good communications management.
One activity that drives this KPA is “the project’s software effort and costs are tracked, and corrective actions are taken as necessary” (SEI, 1993). We have a time tracking tool that the projects use for team members to record their time. The tasks are loaded into the time tracking systems and the team members log their time to the tasks in the schedule. Team members record time to tasks assigned to them and to tasks assigned to another team member that they helped out with. In other words, we expect the team members to record all project related time in the time tracking tool.
After the team members record their time, the Project Manager updates the project schedules. After the information is entered, the schedules are analyzed, and corrective actions are taken. Most of the time some corrective action is needed. We found that the schedule works a lot better if we take many small corrective actions than waiting and taking a few big corrective actions. Most of the time, corrective actions involve switching resources, or encouraging the team member to work a little harder to catch up. Again, most if the time, it’s not a big deal because we are on top of project tracking.
Sometime patterns emerge that our normal corrective actions will not handle. Sometimes our estimates are bad and many small corrective actions will not correct it. In that case, the Project Manager will create a new estimate based on the actual information from the project. The Project Manager will generate a change request and submit it to the change control board for approval. This step can be tough on Project Managers, because of admitting mistakes to the clients. We found that it’s better to start negotiating with the client and get the change approved. In the past many times we ignored the obvious and tried to work a lot of extra hours to make up for the bad estimates. This generated many more problems and most of the time, the project came in late anyway. With bad estimates, there will be unhappy clients. If the Project Manager re-estimates and goes through the change control process, the client will be unhappy, however, the Project Manager has the rest of the project to make the client happy again.
The last activity that I will cover is “the software risks associated with cost, resource, schedule, and technical aspects of the project are tracked” (SEI, 1993). This activity is an important one. The project teams analyze and document risks and create a risk action plan during the project startup and planning stages. This is not a one-time activity. As the project proceeds, the environment changes. As the environment changes, project risks will also change. New risks will be encountered and original risk threats may decrease or be eliminated all together. The Project Manager needs to stay on top of the changing risk threats. A new risk may have as big of a negative consequence as an original risk and that risk needs to be included in the risk management plan.
The Software Project Tracking and Oversight Key Process Area has many other activities that support the management of the project from a Project Management point of view. The above activities had the biggest impact to the management of our projects. None of the above activities are rocket science; however, without discipline, the above activities are forgotten and the project goes out of control. Discipline is critical to good Project Management.
Software Quality Assurance KPA
Speaking of discipline, the Software Quality Assurance KPA is the KPA that helped enforce discipline. The goals of this KPA are:
1. Software quality assurance activities are planned.
2. Adherence of software products and activities to the applicable standards, procedures, and requirements is verified objectively.
3. Affected groups and individuals are informed of software quality assurance activities and results.
4. Noncompliance issues that cannot be resolved within the software project are addressed by senior management (SEI, 1993).
When the process improvement team originally rolled out the CMM Level 2 processes, which included the Project Management processes, the project teams did not follow the processes very well. They were in the middle of providing a product to the client and the last thing they needed was to learn a new way of doing business. This changed as the process improvement team started performing SQA audits. We sent audit teams to inspect the documentation each project team needed to create to be in compliance with the processes. Our first audit results were poor.
The SQA audits showed the project teams that the CMM processes were not the program of the month. We were serious about the processes and management backed us up. During the audits, many of the project teams asked for clarification of some of the processes. The audits turned out to be learning experiences for the project teams. In the beginning, management did not beat the project teams up for poor audit results. Management did expect the audit results to improve over time and the audit results did improve. Now the SQA audits are part of every project and SQA tasks are part of every project schedule.
By the way, we do not force the project team to follow the processes to the letter. We let the Project Manager tailor the process to meet the business needs of the project. We do not force the project teams to fill out documents if it did not make any sense. The last thing we wanted the project to do was to do work that did not add value. We required the project to document the reason why they deviated from the processes.
The CMM processes drive how we do business at the Lansing Solution Center. It has changed our lives for the better. Our success rate for projects has increases dramatically. Unfortunately, it does not guarantee that every project will be successful. Most of the time when a project gets in trouble, it’s because the project team did not follow the processes as well as they should have. Even when a project gets in serious trouble, it does not lead to the long death marches like it did before we reached the CMM Level 2 and Level 3 maturity.
The Software Engineering Institute created the Capability Maturity Model and is part of Carnegie Mellon University. More information on the CMM can be found at www.sei.cmu.edu.
Proceedings of the Project Management Institute Annual Seminars & Symposium
September 7–16, 2000 • Houston, Texas, USA