Participants: Sunil Prashara, PMI President & CEO Sierra Hampton-Simmons, Director & Portfolio Leader, Certification Products
SUNIL: Hi, everyone. Sunil here. And I’m really, really pleased to be back here in Philadelphia [at] GHQ. It’s been a while. I took a few days holiday after my 100-day announcement to everybody. So I took the opportunity to see if we could have another “Straight Talk with Sunil,” and actually what I want to do is introduce a little bit about our transformation and some of the things that we’re doing there.
As you all know, PMI has embarked on quite a significant transformation, which is changing a lot of things in the organization. One of the things that you may have heard about is some changes that are happening to our certifications. So I thought I would take the opportunity to actually find out a little bit more about what’s going on in that space and put somebody on the hot seat. So I have with me here today Sierra [Hampton]. So Sierra has been instrumental in shaping our certifications in the past and taking us forward into the future. So I’m going to double-click a little bit on some of the things that are happening in that space, and hopefully we will both learn a lot more about what’s going on.
I’ve also been given a few challenging questions from some of the REPs that are out there, so I’m going to use the opportunity here to really put her on the hot seat and we’ll figure out where we go from there.
So first of all, welcome, thanks. You’re a very brave girl to come and join us on this thing, and, you know, you are now going to be open to the whole community of PMI, so how does that make you feel?
SIERRA: Ah, it’s exciting.
SUNIL: That’s good.
SIERRA: Yeah. Thanks for having me.
SUNIL:So a big part of our transformation is the certifications. And for many, many years, the PMP has been kind of like structured the way it has been, and now all of a sudden we’re starting to, wanting to change it. What’s driving that?
SIERRA: Well, we’ve been listening to feedback for a number of years from all around the community. A lot of people think that we make these decisions in a silo here, internally at PMI, but that’s not actually true. What happens is, we listen to subject-matter experts who are project managers. So the PMP is being designed by project managers. It’s written by project managers, for project managers. We did some research, and that research was a global practice analysis. It allows us to see the trends that are current today for hiring project managers. What do orgs need? What kinds of skills are they looking for?
And what they told us is that 50 percent of their life is totally about predictive, traditional project management, but the other 50 percent of their life is split between agile and hybrid. And so we really needed to acknowledge and understand that that’s what people really need to manage projects in a modern atmosphere.
SUNIL: So is it like a — are we doing a wholesale transformation? Is it the whole thing changing? It’s that significant?
SUNIL: So it’s completely unrecognizable from [the past]?
SIERRA: Well, let me give you a couple of features. So currently the PMP has five sections of the test that are really around the process of project management, traditional project management. It’s going to three sections. Those three sections are people, process, and business environment. And so in all of those three sections, we’re able to address traditional as well as agile and hybrid methodologies — sprinkled all throughout.
So that’s a very significant change, because previously we were structured just for a waterfall environment. Now we’re structuring ourselves to really kind of contend with the things that you need from a competency perspective, to really be able to manage projects effectively.
SUNIL: What do you see starting to happen as a consequence of these changes? What’s the value that we’re trying to capture here?
SIERRA: Well, one of the things is, it’s going to demonstrate to the world that someone is prepared to really tackle the new types of projects they’re seeing, and they have enough in their toolkits, if you will, to be able to make the decisions about what methodology they want to use.
Previously, when you got a PMP, it said that you could do things waterfall. But what if you’re in an environment where you’re doing technical projects or things of that nature? You need to be able to demonstrate with your credential that you can do them both. And so this is going to allow them to go and say, “Hey, I can lead and direct any kind of project you want to give to me, including those that are a little different — the hybrid ones.”
SUNIL: But one of the questions I got from some of the REPs and some of the volunteers and some of the PMI community was, you know, [the] PMP doesn’t really guarantee you a job, right?
SIERRA: Right. Right.
SUNIL: And there are other skill sets that are required, and sometimes maybe even the certification may not be relevant. I mean, what are your thoughts around that?
SIERRA: Well, it’s interesting. I often have to tell people that, you know, PMP is just one step in a journey of getting a job. What I can say about it is, PMP is designed to identify the minimal skills. So people like to use the PMP as a screening technique, right? So if you’re hiring at scale and you’ve got hundreds of applicants, how do you know this person is telling the truth on their LinkedIn bio profile and their CV? Well, at least with the PMP, you know that they’ve been screened to some extent in their application process outlining their experience across the delivery landscape, right, and they’ve passed a pretty rigorous examination.
So oftentimes, while that may not get you the job, it gets you at the top of the pile so that you can get that consideration you need.
SUNIL: Yeah, because I guess there’s a lot of other skill sets that you require. When you’re in competition for a role and there’s a hundred people applying for the role, you want to put your best foot forward. So I guess it does help because it’s an independent view of your understanding of a particular topic, right, which might be relevant for a role.
It still doesn’t guarantee — I mean, you get a lot of people who are — I mean, I’m a qualified accountant, but you wouldn’t want me to look at your financials. I would get it all wrong, because it’s been such a long time since I was working in finance. So you might get that kind of situation.
How do you — I know we have to do PDUs and things to keep ourselves relevant. Is that going to change in any way or is that staying the same?
SIERRA: That’s not really changing for the PMP. We still require it for people to show that they’re continuing their education, and that’s really important because it says the PMP who got his PMP 15 years ago, as opposed to the person who got it yesterday is keeping current. So, you know, it still has that same meaning, and it really takes the commitment and dedication of the person who holds the credential to say, “Hey, I want to really invest in my career and keep current and be valuable.”
SUNIL: Right. And so how did you know that the changes you made were the changes that needed to be made? I mean, what happens if you got it wrong?
SIERRA: Well, that would be a bad thing, right?
SIERRA: But we really do our due diligence to make sure we get it right. So let me explain to you what this is— I’ve talked about it like a global practice analysis. What that is, is a huge market-research undertaking. We talked to, in focus groups, about 400 subject-matter experts from all across the globe, coming from about 60 different countries and really big, cool companies.
SUNIL: And where did we find these people from?
SIERRA: Well, actually, certification has about 8,000 volunteers who have raised their hands and said, “I want to help with this.” Maybe they got wind of it from friends who’ve helped write exams. Also we get recommendations from big companies. So, Johnson & Johnson, Sony, IBM — they give people up and they say, “Hey, can you allow us to help with this?”
So we’re not just us, looking through our list of people, but it’s also people volunteering on their own and being “volun-told” by their organizations, “Please participate in this.”
SUNIL: And so what do they do? They create questions for you, or they create answers? What do they —
SIERRA: Well, they do a little bit of everything. So, from the beginning of the process, we have them help us design the exam. So they helped us determine how much we should devote to agile, for example, in the PMP. And they help us tell us exactly which topics are important and relevant, and they’re very passionate about those topics.
We take that from there, and we put the survey out to over 2,000 people who responded in five different languages to make sure that what they told us in that room was true across the globe and there weren’t any one-offs. Right?
SUNIL: So to say that slowly: You send it out to 2,000 people, five different languages, to get the input that the questions that have been posed are the right questions.
SIERRA: And the answers are the right answers.
SIERRA: No — we’re surveying on the topics, Sunil. So the topics that get included in the test. And so we surveyed it out to a lot of people, over 10,000 people, but 2,000 responded. And so those 2,000 people basically agreed with everyone we had spoken to — in person, in the focus groups. And that tells us that our design is sound.
From that point on, then we go into item writing, and that is a pretty rigorous process, specifically with the PMP. We do item writing at least four times a year across the globe.
SUNIL: And what is item writing?
SIERRA: Items are questions for the test. And so what we have is we have committees of people, about 400 or so, who are committed to writing items for the PMP. They meet up with us around the globe in small meetings with a hundred people, and they have this one room that’s all about writing the questions and brainstorming new ideas of new types of questions they want to see included.
And then they have this clearing house — the room of review. People sit, and they argue, and they have to come to a consensus across industry, geography, that this question is fair, that it’s valid, and it’s relevant for the test. And once that’s done, then we can release it onto the exam.
It’s a very long, grueling process, but it’s very much so driven by the PMs, the project managers themselves. It’s not PMI driving it — it’s them.
SUNIL: Going forward, then, can you see this activity taking place every two years, three years? Every year?
SIERRA: So the research itself happens every three to five years. But the item writing is ongoing. We do it at least once a quarter, sometimes two times.
SUNIL: Okay, wow. So you’re constantly looking at the relevance of the exam, the questions that sit in the exam, and the way they need to be answered, etc. And that too, international input. Inputs from across the globe giving you a better sense of security that you’re actually asking the right questions and certifying at the right level.
SUNIL: And typically, then, so, for someone who’s coming in new to the world of project management, how long should they really plan before they get their PMP? And is that going to change?
SIERRA: Actually, that answer is “It depends.” Right? If you’re someone who is very senior who’s done this a long time, you may be able to say, “Okay, today I’m going to take my PMP. Let me fill out the application and I’ll go sit the exam and pass it in a couple of days.” Some people are just that lucky and have that much knowledge.
If you’re someone newer, though, our typical candidate is really trying to attempt to get the PMP to show that they’re ready to lead and direct a project. And so they may have a couple passes. It may take them longer to get the contact hours and that sort of thing.
So it depends on the individual.
SUNIL: Can we track any kind of metric — say, typically there’s this number of —
SIERRA: We do but it’s so variable that it really is —
SUNIL: What about pass rates?
SIERRA: We don’t really talk about pass rates, because pass rates are just a snapshot in time. So today the pass rate might be X percentage, but tomorrow the test might get harder, people might study less, and so it changes. And so that’s the reason we don’t share it — because it varies pretty much.
SUNIL: So how do you know where the pass rate is?
SIERRA: Well, we do look at it. We look at it to see the health. But we look beyond that. We actually look at the questions, and the statistics behind them, and see how they’re performing. Are people who should be qualified failing? We do look at that. If that happens, we will reject a question and remove it from the pool.
SUNIL: Right. So it’s not like, I mean my kids just went and did their IB and their A levels, and they say a certain number of people need to get a grade A and a certain number of people need to get a grade B, etc., and then they kind of manage this bell-curve thing, a bit like how they do performance evaluations. Do we have any of that kind of thing going on?
SIERRA: We do have something called psychometrics going on, yes. However, it’s not on a bell curve because it’s not that kind of normalization. But essentially what we’re looking at is to make sure that the test item is doing what it’s supposed to do. There’s all this magic stats in the back. And what we’re looking at is to make sure that people who are high-performing can do well, and people who have shown us that they’re low-performing aren’t doing so well. So if that’s consistent and reliable, we’re confident that the test is sound.
SUNIL: The challenging question I have in my head is: If it’s too easy and everybody passes, then it doesn’t have the value.
SIERRA: Right. It’s not even worth it.
SUNIL: If it’s too hard, and people are very busy at day-to-day jobs, they’re not going to find the time to do it. So how do you find the right balance? Because you’re still controlling this with your team. How do you know you’re at the right level of the right number of people coming through so that you don’t dilute the thing and say, okay, well, every Tom, Dick and Harry’s got one so it doesn’t have any meaning, or, you know, this is so difficult to do, I don’t have time to do it?
SIERRA: Well, actually, there is a process for that. After we’ve written an exam, we do this thing called a standard setting, which, again, is volunteer-led. So we have thought leaders who come and look at each question that’s been presented on the test, how it’s done statistically, and they give us feedback — out of 100 people who meet our target audience, 70 of them will pass this, 80 will pass this, 90 will pass this. Only two would pass this!
And using that data, we add it up and we determine if this is indeed a healthy exam, and we figure out the cut mark. That is the mark that you have to get, the minimum score you need to get to pass. And that’s what we do for every exam.
SUNIL: So I’ve had a few people call me and say, “You know what? I worked really, really hard, I’ve had 20 years experience in project management — I failed by one grade, one point.” Is there not something you can do to help me cross over that, or do I have to sit the whole thing again?
SIERRA: And they have to. Yeah.
SUNIL: Okay. It’s a bit like when I did my CMA exams - if you fail, you’ve got to sit all the exams again, and do it all again. So that’s an important point, because though I do get a lot of e-mails from people who say, “I worked really, really hard and I failed by just one point in a particular area.” And they ask for a remark and it gets remarked, and yeah, you know, they failed by that one point. And, you know, there is a line that you have to draw. There’s no gray zone, right? Is that what you’re saying?
SIERRA: Right. So we do have — when we set that mark, one of the things we’re looking at is standard errors, right? And so we do account for that. We have a buffer zone around that. So if I tell you, you need 50 out of 100, perhaps we’ll set it at 48 to account for those standard errors. So that guy who fails by one point, unfortunately he does have to sit for it again, because in order for us to be fair, we can’t just give him that point, right? He has to go through the questions again and holistically be able to pass it. And we try to count and be fair by adding in that buffer zone because we know that.
SUNIL: So arguably when they’re on that borderline anyway, by one or two points, it is, you know — you could argue that they’re just barely passing.
SUNIL: As opposed to someone who’s a slam-dunk pass. Okay, he’s got it.
SIERRA: Right. Well, it’s a pass-fail instrument. It’s not designed to say, here’s the top of the heap and here’s the bottom of the heap. It’s really designed to say, the person knows it or he doesn’t know it, and here’s the lowest mark that we can say that they know it. And if they’re one below, they don’t know it. They need to come back and take the test again.
SUNIL: We’re changing.
SUNIL: And you explained very clearly why we’re changing and how we’re changing and what the process has been to help us figure out what the changes should be. The next stage, really, is to get this out into the community. And that is by making sure that the REPs and all the folks that actually run training courses in PMP and project management are aware of what those changes are. Are you involved in that part of the process as well?
SIERRA: Yes. Intimately. Very much so.
SUNIL: So how are you making sure that everybody knows what those changes are and that they can train at a level that they need to be able to?
SIERRA: Yeah. So, we have to first address one thing. So in the past, when PMP changed, the change was minimal. It was a few topics added, a few topics scrubbed away. But now we’re having this drastic, even structural change of the examination, and we realize that this is an impact.
And so what we have done is: first of all we’ve been very transparent with the change. We have a micro-site that tells people what the exam count and outline is, and it has all the topics there. In addition to that, we’re partnering with the folks in professional development as well as the folks in the REP team here, to be able to support the community in making the changes they need to make to the training. Because ultimately, our goal is we really want to put out a good exam, but we don’t want to put something out that people aren’t prepared for, so we need to make sure that our partners — the trainers, the educators — are aware, can interpret and adjust to this change in the right time to really meet our customers’ needs.
So we’re doing things like webinars to REPs and other kinds of documents, and we’re really listening out to: What do they need? What is missing? And how can we help them? And that’s our intention.
SUNIL: So when I think about it, the content of the exam and the toughness of the exam is one thing. Then the quality of your teachers is another thing that can influence. And then the location. And then also the way you’re taught, you know? I mean, there are different ways — you can have face-to-face, you can do different [methods]. These are all differences, and it’s ebbing and flowing and changing. Are you involved in all of those?
SIERRA: No, I’m not, actually. As a person who manages the certification team, because of conflicts of interest, I can’t get too involved in any of that.
SUNIL: Okay. That’s actually what I was alluding to.
SIERRA: Right. So the things that we are doing, because of the change in the exam, is more of about awareness and, really, education to the changes, so that people understand why we changed, how it changed, how that will impact the exam.
SUNIL: And what’s changed.
SIERRA: Right. Exactly. It’s not me telling people how to train them, how they should reform their curriculums, what strategies would work. That would be inappropriate and cross the line. And so I’m not doing that.
SUNIL: So you have a Chinese wall, there, right?
SIERRA: We definitely do. Definitely.
SUNIL: Tell me about Pearson VUE. What is that? That’s something new that your organization has created a contract with. What do they do?
SIERRA: Pearson VUE is a test-delivery provider. We changed to them on July 1 to deliver all of our exams across the world with the exception of mainland China.
SUNIL: So when you say they deliver our exams, what do they actually do?
SIERRA: So a candidate goes to their web site, registers for the examination, and then goes to a facility that is branded as Pearson VUE, and they sit down at a computer and take the test. Or they take it from home, depending on which test they’re taking.
SUNIL: Okay, I got it. So this is the organization that actually conducts, creates the environment for the test, and then captures all of the input. And then, are they the ones who also mark, or —
SIERRA: Yes, they mark it. Their systems mark it based upon the inputs that we gave to them. So we give them the scoring logic.
SUNIL: And the questions, are they multiple-choice or what are they?
SIERRA: Currently our exams are multiple-choice. But one of the things that we are going to do this year, actually, is update our exams to be more digitally enhanced. You will see things like moving graphics and other media. We’re looking at animations and all of those kinds of things. And those are things that we are getting out of this relationship with Pearson VUE. Our other vendor could do some of the same things as well; however, we’re able to do it natively with some of the tools we have here in-house and get them out to the market pretty fast.
We’re really excited about the Pearson VUE launch because it’s really allowing us to do a lot of other things that we haven’t been able to do. One of the big things that’s near and dear to my heart is being able to localize the experience for our candidates. So after they’ve been accepted and are eligible to take the test, they’re able to register on a Pearson VUE site.
So in Japan right now, when a candidate registers, the site, the web site at Pearson VUE, is all in Japanese for them, including the directions and all of that stuff. If they happen to be an expat and really want it in English, they can toggle back very easily to English.
In addition to that, when they sit at the test, we’ve also enhanced our items — our questions — so that people who are taking the test not in English have a better experience. Previously, we used this thing called a language aid, which was essentially like a split screen. So imagine, on the top of your screen you have Japanese; on the bottom of your screen you have the same question in English, and they have to answer in that English, and all the interface is in English. Well, that was a little — you can imagine — not so easy.
So what we’ve done is we’ve gone to one single language, so that candidate in Japan, in Tokyo, they get that exam question in Japanese. The whole user interface is now in Japanese, their help text is now in Japanese, tutorial is in Japanese, and even the surveys are in Japanese. And the communications that they get from Pearson VUE will be in Japanese as well.
SUNIL: That’s all great. But I got stuck on one point earlier on, so I’m going to drill a little bit on that. You said that it’s multiple-choice, which means an either-or type of answer. And I’ve run loads of programs in my life before, and often it’s never an either-or answer. You’re like in the gray zone, and you have to read so many other things — like politics, the stakeholders, the timing, culture — to make a decision on slowing down a program, adding additional resources, not meeting a certain milestone, or whatever, for the bigger and better cause or something else.
SUNIL: How do you assess, in a multiple-choice way, that complexity that you have in projects?
SIERRA: So, it’s important to note that we are changing from just having multiple-choice, single-response. So that’s important to note.
But in today’s world, that is the exam. And the item-writing process that I spoke to you about earlier, that’s what those review sessions are about. So you have teams of subject-matter experts who really challenge each option, and they make sure that everything that’s not the answer, that’s not noted as the answer, is incorrect.
SUNIL: But say if someone’s writing some verbiage, then how do you know that Pearson VUE have got the skill sets to be able to turn around and say, okay, that’s a good answer and that’s not a good answer?
SIERRA: Well, Pearson VUE doesn’t determine that; PMI does, and we just publish through them. So they have to publish what we’ve given them.
SUNIL: Okay, I got it. So we will have provided what we believe are the answers, if it’s multiple-choice, and if in the future it’s not multiple-choice, again, Pearson VUE will collect the answer, but now they will then send to us and we’ll decide whether that — if it’s an essay they’ve written, for example — are we going to decide whether that’s a pass or fail, or is Pearson going to decide?
SIERRA: So we always decide the answers to our exams, and I would never suggest for us to do anything that was essay-driven stuff, because that’s too subjective. So some of the things we’re looking to do in the future are: automate it, so it’s using kind of like artificial intelligence to score it in the background, and scoring logics and scripts.
SUNIL: Right. Are we doing any of that?
SIERRA: We’re going to start doing that at the end of the year, a lot of the things that we’re doing.
So it’s a programmatic kind of thing. So in the background, there’s a program that’s running that’s scoring it and checking the answers for them. We give all of the intelligence that is being used for that to Pearson VUE and Pearson VUE just implements it for us. It’s not like they’re making the determination. Even with the new item types that we’re going to do, we’re going to give them the scoring logic.
SUNIL: So I think we’re coming up very close to the end. I’ve got a couple more questions. [The interview has been] very, very informative, really good. I’ve got a lot of assurance for myself personally that it’s being structured in a very constructive way. We’ve now incorporated agile into our technologies and thinking, etc. That’s the way the world is going. What do you think is going to come next?
SIERRA: Well, we’ve been doing a bunch of research, and there’s a couple of hot topics. One is strategic project management, so we’re looking at that because everybody wants to be strategic and what does that mean? So we need to be able to help them answer that.
Another one that I found very interesting is that a lot of project managers want to become product managers. And how might they bridge that and move into those careers? Not just product from an agile perspective, but actual true product management from a corporate perspective, from initiation to grave for a product. So that’s very interesting as well.
And then there’s all kinds of interest in other things, like design thinking and things of those natures. So I’m thinking that, from a long-term perspective, we can’t create big PMPs on those kind of things, right? We’re going to have to look at certification a little bit differently. Maybe in some cases, someone has a PMP and then they add a micro-credential on some other topic that allows them to specialize with their organizations and really stand out and say, hey, I’m an expert on this.
Perhaps we even get beyond just testing to other things, like writing white papers and becoming thought leaders and we backing that up. There’s all kinds of ideas that we’ve researched, and we’re still in the infancy stage there, but look for changes because we really want to respond to the market needs and support all of these folks who have become a PMP. As they continue on their career journey, we don’t want them to necessarily walk away from us — we want to support them through that.
SUNIL: That’s fantastic.
SUNIL: I think we’re out of time. That was a fantastic session. I really learned a lot. I hope the audience have also learned a lot. If you have any questions, please flick them over to myself. You should all have my e-mail by now, and if you haven’t, it’s uh ... What is my e-mail? I’ve forgotten what my e-mail is! [Laughter.] It’s [email protected] [Laughs.] I was tap-dancing there. Sorry! So I mean, you might just want to keep this rolling. So what was it again? [email protected] So, drop your e-mails there.
Loved this session. This was supposed to be real “Straight Talk” hard, but you answered the questions so eloquently, so clearly. Seriously, well done. Thank you very much. We should have you on here again.