Show me the money
a panel of experts dissects popular notions of measuring project management maturity
At what level is your organization? And on which of the several models floating around out there? And does it really matter? PM Network asks the experts and gets some surprising opinions about maturity modeling.
by Jeannette Cabanis
IT'S NO SECRET that some organizations are better at project management than others. It's also obvious—at least to the audience of this magazine—that those organizations that do project management well have a distinct competitive advantage. But as a relatively young profession, project management is still groping for the quantitative proof of its usefulness as a business strategy.
As the leading professional association for project managers, PMI is actively working to help design the tools with which project managers can measure and improve the discipline's effectiveness. One of those tools is the maturity model, currently the subject of intense effort by PMI's Standards Committee (see “Standards Committee Tackles Project Management Maturity Models,” by Margaret W. Combe, PM Network, August 1998, p. 21).
In considering how to frame an assessment of project management maturity (PMM), project managers have been tantalized by the Software Engineering Institute's Capability Maturity Model (SEI CMM) for software development organizations. The SEI CMM, after all, does point out project management as a key process area in gauging an organization's capabilities. Yet, for non-IT organizations, some of the SEI model doesn't apply; and because it is focused on software development, not project management per se, it doesn't drill down to the level of detail that would make it truly useful as a measurement tool.
Nevertheless, the SEI's five-level rating of organizations has so far defined attempts by individuals and private companies to define project management maturity. But is that the best way to approach it? And once you've got a model in place—to be bold—so what? How does gaining a high rating against the model translate into bucks on the bottom line?
PM Network decided to collect a handful of subject matter experts on project management maturity to compare notes and mull over some of these questions. At the Project Leadership Conference in San Francisco, on 17 June 1998, our four SMEs shared experiences among themselves and with a small audience drawn from attendees of the annual ABT Corp.-sponsored IT project management event. Here are some highlights from their conversations.
PM Network: We've asked the four major participants to talk about project management maturity from their various perspectives. Cathy Tonne, of Integrated Project Systems, was on the research team that produced the PMI study on the financial and organizational benefits of project management, which made use of the IPS-developed maturity model. She's also currently on PMI's R&D Committee.
Tonne: [H]ow do you improve your business performance? How can project management help facilitate that? … It's this driving, burning need [people] have in their business that is necessitating them requesting help in the project management area. [But] if you implement project management in an organization that doesn't have a supportive environment for that implementation, people are just extremely frustrated and you don't get the benefits.
In implementing project management within an organization, we find it's evolutionary—it's not a one-shot deal. People can't seem to swallow a whole elephant at once, so we need to do it in a stepped process, continually make improvements, measure that, and make adjustments.
The IPS PMM Process was developed, like many other models, based upon the SEI CMM: five levels, six key process areas. Like [project management implementation], it's an iterative process. … In talking about project management maturity, it's really a process, not just a model. It's all the steps, all the activities that need to go into the PMM process that they're working on with the model as a framework.
[IPS] got involved in the PMI study because we didn't have information on why project management is really important to the business in terms of numbers and quantifiable data. Mostly what we had was anecdotal information. And the management and senior execs need to understand, because they are making a pretty significant investment sometimes, what is their return on that investment, for implementing project management within their organization?
We see this study as a first step. [It] was a very grassroots effort. In the four years since the study was started, PMI has grown phenomenally. There's infrastructure now in place for PMI to start working on developing a standard and/or a model for maturity much like SEI did theirs. There's a lot of activity in this arena.
PM Network: Tom Block is our expert on the project office and the role that plays in PMM.
Block: I run into problems [because] we don't have a standard maturity model. … Thus, there are a few companies that are trying to beat the rest of the competition to the street with a maturity model, and that causes confusion for a lot of folks because you're not sure what the point of reference is.
For example, I've just received an RFP and the company said they assumed they were at Level 3 and wanted to get to Level 4 on some unspecified Project Management Maturity Model. [But] when they showed a list of issues that they were trying to solve, you would think that they are probably at Level 1 … but against what standard?
[Then, too] companies are doing this SEI CMM, even though it really doesn't matter to the customer. For example, a large insurance company is trying to get to CMM Level 3 when clearly the buying customer couldn't care less whether the IT office is at Level 1 or Level 5. It doesn't matter, as long as customer service for that insurance is being satisfied.
[T]o get to a level is easy, but to maintain that level over a period of time and to transfer that maturity or knowledge over many, many projects is a long-term effort. … The Project Office (PO) is to my mind that anchor in the organization that allows us to proliferate project management knowledge to other organizations. You see a lot of companies that want to get to Level 2 or Level 3 but they don't want to enforce the discipline to maintain the projects. And so it's almost a self-defeating thing.
Over the long haul, with a process in place and with some discipline enforced by senior management, I think an organization could improve their PMM performance. But the point is not that “I'm at Level 3,” [it's] do you do something that makes the business more successful? I don't care if they're Level 2 or Level 5, if they can cut that cycle time down, that's a big business impact that the operations people and senior management are going to attract to, not to a Level 3 or Level 4. What I hate to see happen is that we're going through this for it's own sake: project management for project management's sake.
PM Network: Richard Bauhaus is Hewlett-Packard's manager of their Project Management Initiative.
Bauhaus: I would state that we've done a pretty good job in terms of project management at HP, as evidenced by our success, but I would have to say we are a Level 1 organization.
Why do I say that? Because we don't have repeatable processes. What we have is what I call heroics, which work very well when you have individuals that have the charisma, the intelligence and the innate ability to make a project work. But could you repeat that person's effort with another person? No. And that's the dilemma that we're in.
We've got a lot of new people who are being asked to be project managers, and they don't know anything about what that means.
[O]ur start [in HP's Project Management Initiative] was, naively, “let's go out and train project managers.” And, I'm here to tell ya, it doesn't work. Not that training isn't important, but just training will not do the job. We discovered that early on. So [here's] what we've learned has to be done in order to actually make an organizational change—because that's what we're in the business of doing. We are actually changing our organizations to think PM methodology in order to achieve those new product developments, those support or service changes, those process changes internally, those organizational changes.
First of all, I have to say, we have a written purpose. It is to dramatically improve our local business execution through rigorous application of project management practices. It is business related. We aren't there for project management's sake, as Tom was talking about, we are there because we need to make a difference in the business. We are trying to do this to gain a competitive edge. We want repeatable processes, we want predictable schedules, we want credible resource management.
We are focusing a lot on software development process because a lot of our projects are that way, whether service/support, new processes or NPD. And we have got mapped what we do in the way of good project management processes. When you talk about Level 2 [of the SEI CMM], the six key process areas (KPAs), five of them map directly to solid project management process steps [Requirements Management, Project Planning, Tracking and Oversight, Subcontract Management, Configuration Management]. One of them is an indirect mapping and that's the quality assurance one.
So you can see, the real hump you need to get over if you are looking at an SEI-type capability maturity model, is Level 2. … I think the SEI CMM model, even though it's for software, really applies to software, hardware, support and services and I think pretty well dictates that, if you have all the processes that are talked of in the five KPAs that are directly related in Level 2, you've made a huge step.
Now, the study that Cathy was just talking about is really interesting but it's insufficient. We are hoping to see that carried a little bit further because it should be able to tell us, what is it worth to be able to get from Level 2 to 3 or 3 to 4 or 4 to 5. Is it a reasonable business investment? It may not be. Some of us feel like maybe it's not worth it. Maybe Level 3 is enough. Maybe Level 2 is enough.
These divisions are spending on the order of a quarter of a million to half a million, maybe more, dollars to implement new, good project management methodologies. So we have got to build a better case for the ROI for becoming a good project management entity.
Block: [That's] a good point. Why go up this ladder if we can cut our cycle time and have quality outputs? That's really all you need. It's that jumping-on-the-bandwagon syndrome that I hate to see, because we're going to end up not being successful because people are doing it for the wrong reasons.
Tonne: You can use ISO as an example, where people just jumped on the ISO bandwagon and there wasn't really a business driver to do that. And now there are organizations who are just choosing not to recertify. They don't want to do this for an outside entity. They'd rather focus on their internal needs, and the business drivers, again—and they don't want to do that with maturity either.
PM Network: ABT has a slightly different spin on maturity modeling. Edward Farrelly is the vice president of product development for ABT Corp.
Farrelly: [As] a software company, we're vitally interested in this whole topic. Most of our business is in the IT project management segment, which is characterized by immature processes—heroic efforts, faddish project management ideas … that come and go. [A]bout five years ago we began to see this lack of maturity as a constraint on our business. How could we help accelerate the maturity of IT project management, how could we get organizations to be more successful so that they would buy more of our stuff?
Since we're a software company, we started with a technology focus … we'd go out and we'd talk to the IT executives and the program managers and they'd tell us about the need for measurement, the need to collect information, so we came along with some metrics and measurement technology that looked at things like productivity metrics and quality metrics and customer satisfaction. Then we sat back and waited for the money to roll in … it didn't happen, and as we tried to drill down into what the disconnect was, with virtually all the senior management saying “This is very important” and virtually none of the operational management getting on the bandwagon, we began to realize it was the lack of a framework, the lack of any sort of theoretical basis for which you could find information that was worth the effort of capturing, synthesizing, and feeding back to recalibrate the process. That was a significant missing link.
So we put the technology-centric approach aside and came up with this thing called the PMM Evaluation. We borrowed very heavily from the PMBOK™ Guide: scope and time management and communications, and so forth.
Reader Service Number 5027
Want to read more?
Check out your back issues of PM Network and PMI's online bookstore at www.pmi.org for these titles:
The Benefits of Project Management: Financial and Organizational Rewards to Corporations, by William Ibbs and Young Hoon Kwak, (PMI, 1997).
Two articles based on this research have also been published in PM Network: “Measuring Project Management Return on Investment” (PM Network, November 1997, 36–38) and “Benchmarking Project Management Organizations” (PM Network, February 1998, 49–53).
Material on best practices and key process areas resulting from the Fortune 500 Benchmarking Forums was published as Best Practices of Project Management Groups in Large Functional Organizations, by Frank Toney and Ray Powers (PMI, 1997) and as “What the Fortune 500 Know About PM Best Practices” by Frank Toney, PM Network, February 1997, 30–34.
Another maturity model, based on one originally developed by Micro Frame Technologies, is discussed on the Web at www.pm3.com and was described in “Adding Focus to Improvement Efforts with PM3” by Ron Remy, PM Network, July 1997, 43–47.
PM Network columnist Paul Dinsmore gave an overview on “How Grown-up is Your Organization?” in his June 1998 “Up & Down the Organization” column. ■
Our first take was, why don't we just use the CMM; that has a lot to do with project management, let's see what we can apply there. We went out to our client base and did some surveys and found everybody out there claiming they were Level 3 or 4 and our research came back and most of the clients were anywhere from a negative 1 to a .1. You'd find one project that really had their act together, but on an organization basis, [it was] relatively weak, no sharing, no consistency, so you couldn't transfer that knowledge. It was heroics.
So we recalibrated. We said, “If the whole world is in this narrow band at the lower level, we'll define our own 0–5 scale that maps against the CMM 0–2 because you need that level of granularity to be able to give people some success.” It's demotivational to go in and say, you are making things worse … we decided at least everybody gets a positive grade. And the service that we evolved was relatively simple.
We would go in and, through questionnaires and interviews at all levels with all stakeholders, adapt a series of profiles: what do the users think, what do the providers think, what do the project managers think, and so on, and do some crossanalysis to spot the vast discrepancies, as it turned out, between what the project managers thought was important, what the people who were paying for this stuff thought was important. What we'd do was to establish an initial baseline level of maturity and we would also, based on the interviewing process, establish a target range. This gets to the point Tom was raising—you don't have to be excellent in everything; so where do you think you need to be, for your organization, in a particular competency like risk management.
With those two measures we'd do a gap analysis, and out of that would come some measures that would get acted on. The idea was to periodically revisit this, rebaseline, see how we're doing, and recalibrate. We began to do some of these PMMEs … and interesting results turned out. For one thing, the project management professionals always had a significantly different view of priorities and of the level that we needed to get to versus the line of business and executive management. So right away people were not aligned.
[W]hen it came to picking the priority areas, the business people and the execs tended to be very, very concerned with risk and with cost and those ranked lowest on the IT PM's scale, so there was a real series of disconnects as to what mattered the most.
We've done some [PMMEs] now, [and] there's been a little disappointment in terms of the actual results … we've had some people who ranked relatively high on the scale and made lots of progress but there doesn't seem to be a strong correlation between people that set out to become more mature and whether or not that becomes a successful organization or whether they get outsourced because they aren't actually delivering results.
So that took us to the next evolution. A lot of organizations have come to us and said, We've decided that there was too much emphasis on project management that was inward-focused—project management for its own sake—we need results. We want to be focused on outcomes. This is probably specific to the IT segment, but we are seeing an awful lot of IT shops saying, we'd like to reconfigure ourselves more along the line of software vendors and to treat things as products, to manage the process from a configuration and release management standpoint—from a product management as opposed to a project management standpoint.
The interesting news here—which is anecdotal at this point because its relatively recent and not based on a vast number of people—is that people who are adopting that kind of a focus seem to be gaining the benefits that they wanted—more productivity, reduced cycle times, and so on.
PM Network: Going back to this idea of an industry-standard maturity model: There's been so much competition around this issue in the last year, even rumors of corporate espionage, which seems really counterproductive. Do you have any thoughts on how we can overcome this competitive atmosphere and move toward collaboration, which seems important for the profession?
Bauhaus: We have a foremost expert on the SEI model here [in the audience]—how did you do it there, Gene?
Bounds: I'm Gene Bounds, and I mentioned to Richard earlier that I took a sabbatical from my company in 1991 and went to the SEI to learn more about the process management part of being a software engineer. My company said, okay, 2 years, go do it. Seven years later, I came back, having worked with [the SEI] team to pilot, apply, and improve the software model. … What SEI did, was to become the steward. They called 400 or 500 people together to say, what are the best practices? [T]hey had enough buy-in that they would say, we can live with that. So it isn't as though SEI created this thing and it became a standard. They actually pulled people together and helped to evolve it.
Tonne: The other thing about the SEI [is that] they have statistics and are able to tell you what it's worth to get to the next level. And that's a really big selling point.
Bounds: In order to get to that point, you must first have the best practices, and then also get data back. That takes about three years. It was three years after the CMM version 1.1 came out that we were actually able to get data. [Y]ou have to be patient.
Tonne: [T]here's a double-edged sword in doing more study on the value of project management. What if we find out there is no value? People know we need to do this, but there's still that question. Within the organizations that were studied in [the PMI] study, there was a diminishing return as you got to the higher levels of maturity, and so as you mentioned before, it just may not make sense for some companies to get higher than 3. Although I am hearing from people, I'm a Level 4 or I'm going for Level 5 … it's almost as though they've lost sight of what the real purpose is, which is improving your business performance.
Farrelly: What we almost always hear from the business side is that, once you've got things predictable and repeatable, that's as far as we want to go, the rest is esoteric … they don't want to get involved in that.
PM Network: Is that level of diminishing returns related to the size of projects that a company normally engages in? For example, maybe if you are mainly involved in small, shorter-term projects you only need to be Level 2, but if you habitually engage in multimillion-dollar extravaganzas involving a cast of thousands worldwide, then you need to be at higher maturity level. Today there's a movement toward smaller projects, shorter time frames … if you look at the recant CHAOS Report studies … projects are getting better and at the same time we're moving toward shorter time frames, phased delivery. Are things getting better because project management is getting better or are things getting better simply because we've changed the strategies so we don't need to be as good at project management?
Louis Leatham: Doesn't that depend on the criticality of the project? Statistically this happens when you are … wanting failsafe and knowing the last 3 percent perfection could costs as much as the first 97 percent … whether your project is grinding beans for Starbucks or working on the space shuttle may determine that level of required expertise.
Bauhaus: To me, it's not the complexity of the project that dictates higher-level process performance, it's the complexity of the organization. The higher levels at 4 and 5 are really talking about organization learning issues. Maintaining an organizational memory about how you do things is very beneficial.
PM Network would like to thank the panel members: Richard Bauhaus, manager, Hewlett-Packard Company Project Management Initiative; Tom Block, PMP, of Project Solutions Inc.; Edward Farrelly, ABT Corp.'s vice president of product management; Catherine L. Tonne, PMP, Integrated Project Services, for sharing their expertise. Thanks also to the audience members who participated in the discussion: Eugene C. Bounds, PMP, vice president of the PM Solutions Group at Robbins-Gioia Inc., and Louis Leatham, project manager, Operations Engineering Information & Support Services, Boeing Company. We are indebted to ABT's Shannon Mallozzi for coordinating the event.
One of the issues that we find is, we have very mobile population inside of HP. [I]f you have project teams that are working in Singapore, Fort Collins, Grenoble and Bangalore, you want them all to have similar process understanding so they can talk to each other, so they can make the project work. … To me, the higher levels really talk about maintaining a more standard position with respect to the whole organization.
Tonne: [W]e found a lot of leverage in the utilization of a common framework and a common vocabulary. When you have people who are moving from project to project, having a similar framework and common vocabulary, you can make that transition a little easier.
Bauhaus: [Also] if we are working with a project that has many pieces, many vendors, or other groups inside HP working with each other, they need that common framework so they can communicate with each other, so you can manage the interfaces.
Farrelly: That's the other implication of these smaller projects, that they have to aggregate … there's a lot more interproject communications and dependencies at that point … you can export some of the issues but they are still there.
Tonne: [T]here's more rapid opportunities for self-correction when you are going through more projects in shorter time frames. You can make your adjustments and don't get quite as out of whack as the longer ones.
Bounds: There's another important lesson here. Projects rarely worry about maturity—organizations do. In fact, if they could do it over again, the SEI would have no levels. They were forced into levels by the government, who wanted to use this model to award contracts; they wanted to say, we are going to award this contract to the organization that's most mature. To the SEI what's important is are you satisfying the practices at a level or no? Projects want to satisfy the practices that are in your model because it helps meet the objectives and be repeatable. It's the organization that's focused on the notion of levels.
Reader Service Number 5133
I wish that PMI would find a way to diminish this notion of levels and focus more on what are the key practices, are they important to our project, and how we can best satisfy them.
PM Network: If we weren't going to measure maturity by levels, how would we do that? What would this look like?
Bounds: I don't think they'll be able to avoid it. But I'm just hoping that at the project level they provide a model and framework that allows us to look at the practices that are required. And then at the project level what we get into is that we either are or are not satisfying those best practices. It's when you get to the organization level that I think it's going to be important to have relative measures. We aren't going to be able to avoid it … I'm just hoping that it won't be the primary focus.
Leatham: Richard, you said that you thought your group was at a Level 1 … Is that where you want it to be?
Leatham: So the different levels allow you to target different chunks of improvement that you want to achieve. I think if we took my group at Level 1 and said, we're going to Level 5, we'd never get there, people would give up. So the levels can serve that purpose, as far as improving.
Bauhaus: There's another way you can approach it though. If you counted all the KPAs, you could say there are 23 competencies or best practices and you want to reach for 10 percent or 20 percent of them. But you still get into what are the key ones, what are the first set you need to have, what are the second set you need to have.
Bounds: If I say I'm going from ad hoc to planned, what we want is to get our projects to satisfy KPAs at those levels—not focus on the levels initially. After you've gotten to where you are satisfying those KPAs, then I think you can introduce the notion of levels. Until we get those fundamental building blocks in place at that planned level I think all the numbers are just mystique.
Leatham: Is there a timeframe for PMI to produce something that we can use to help us in our industry? At the Boeing Company, we have 2,000 projects going on, we can't get project managers, we're dealing with a hard resource to procure, so we really need some help in this. [N]ot having a unified approach, there's nothing available for us.
Tonne: That's the next milestone. … One of the things that the study group heard was that PMI could either establish, from among the existing models, a standard to which all of us working in the industry can work, or go ahead and develop their own model. Currently they're evaluating developing their own model.
Bounds: If there was ever a role for PMI to play, it would be as that steward to get something in maturity that project management users can understand. Until you do that, we aren't going to have anything to measure from.
BUSINESS TOOL OR academic exercise? Our panelists' consensus seems to be that the maturity model, like any other tool, is only valuable if and when it's the right tool for the right job. For project management maturity modeling to fulfill its promise, the right tool—a standardized model—must be used for the right job—improving business performance, as measured on the bottom line.
One thing's certain: the conversation about project management maturity is far from over. In fact, it will flower anew on Sunday, 11 October, in Long Beach, California, at the PMI Standards Committee working session just prior to the Annual Seminars & Symposium. To register for this free day-long session, please e-mail Lew Gedansky at firstname.lastname@example.org. ■
Jeannette Cabanis (email@example.com) is special projects editor for PMI Publishing Division.
PM Network • September 1998
PMI research shows project teams that draw from an array of perspectives and skillsets deliver powerful outcomes.