Abstract
This paper describes the added value of using gamification in project management education. The purpose of this paper is to evaluate the advantages of gamified project management courses versus the classic ex cathedra set up. After an introduction to the concept of gamification, this paper evaluates an actual case study of a gamified PMBOK-based project management course for junior project managers in Europe. The evaluation is done using the four-level Kirkpatrick evaluation model for training, evaluating the reactions of the participants, the level of cognitive learning, and the way actual behavior changed and finally, the return on investment of this gamified course.
Introduction
In this paper I will explain the basic concepts of gamification and describe an actual case in which gamification is applied within the area of project management education. I will compare the gamified course (“serious game”) to the standard classroom set up. I will point out the differences and the added value of the use of gamification, based on actual findings gathered for more than 10 years of the company using gamification in their project management course. The evaluation of the gamified course will be done on four levels, using the Kirkpatrick model for evaluating training.
Introduction to Gamification
Gamification is the application of design concepts and techniques, loyalty programs and behavioral economics in a business environment (Zichermann, 2013, p. xii). Gamification is often applied in the area of user engagement or customer engagement and training. Gamification is a fairly new trend. It only showed up as a Google Trending topic since 2010 (Zichermann, 2013, p. xii). Recently, the research institute Gartner positioned gamification in the hype cycle for emerging technologies at the top of the “inflated expectations” phase (Exhibit 1).
Exhibit 1 – Hype Cycle 2012 for Emerging Technologies from Gartner
Recent studies underpin the problem why Gamification is at the peak of inflated expectations. Gartner (2012, Paragraph 2) predicts that 80 of current gamification projects will have failed by 2014. The reason why the large majority of all current projects will fail is due to the approach applied, in other words the game-experience is badly designed. These failures will probably slow down the adaption process of gamification, but Gartner projects gamification to be fully incorporated in daily business life within 5 to 10 years (“Plateau of productivity”). The long-term perspective for gamification remains positive. Gartner estimates that 70% all the world's largest organizations will be using gamification by 2015 (Gartner, 2011, first paragraph). Other market researchers predict a flourishing future for gamification. M2 Research forecasts (Meloni & Gruener, 2012, p. 7) that U.S. companies will almost double their yearly in investments year by year, spending 2.8 billion U.S. dollars per year on gamification technologies and services by 2016 (Exhibit 2).
Exhibit 2 – Actual & Projected Spending on Gamification Technologies and Services
Serious Game Versus Gamification
Before we get to the example, a distinction between gamification and serious games need to be made. Serious games and gamification have in common that they both use game design and game elements (Marczewski, 2013) and they both serve a business purpose: increasing employee of customer engagement, improving the learning curve in education…The main difference between gamification and serious games is that gamification is not using gameplay where serious games do. Some of the most well-known examples of serious games are Plantville from Siemens (a serious game focused on educating plant management) and “Pass It On” from AXA Insurance (a serious game focused on personal financial planning) (Marczewski, 2013; AXA, 2011).
In my personal opinion, serious games should be viewed as a subcategory of the total gamification initiative, focusing on learning and growth.
In the next section I will elaborate on an actual case study, a serious game I developed to improve the education of project managers.
Euro Wonderland Case Study from CTG, a Serious Game on Project Management
More than 10 years ago, as the leader of Competence Center for Project Management in my company, I abandoned the classical classroom setting in which project management courses were organized. Since then I have gamified the project management training courses at CTG (www.ctg.com) by creating a serious game, named Euro Wonderland. In the next sections I will first explain the concepts and set up of this gamified course and then I will evaluate this gamified project management course compared to a traditional way of providing training.
Serious Game: Euro Wonderland
For more than 10 years, junior project managers at CTG Europe, a IT services company, are invited for a two-day project management course, organized not in the professionally equipped training center of our European headquarters in Brussels, but in a modest community meeting center in a very small town, about 100 km east of the Belgian capital. This room serves as the “war room” from which all project management (training) activities are executed (simulated), but part of the action takes place in a national park in which the small village is located. This is the place where eight to maximum 12 junior project managers survive their first real-life project. The project they need to manage is the startup of an entertainment park in the specific national park.
Through role-playing, realistic scenarios and typical project stakeholder characters, they need to plan and implement the entertainment park within budget, in time, and within scope. During the game, situations changes (a protest from ecologists, owner wants a new ride for his park) as in real-life projects change. The game simulates these project changes to learn them to deal appropriately with change management in real life. In the same way other knowledge areas from the PMBOK® Guide are touched, demonstrated, and exercised by the participants. This serious game is focused on exercising the application of basic project management techniques and deliverables such as work breakdown, task list, network scheduling,…using the latest PMBOK® Guide as the guiding principle and common language.
The PMBOK® Guide from PMI is used since it is the basis of the company's project management methodology, but also because it's becoming more and more the dominant market standard for managing projects in Europe. The focus in this gamified course is not on the theory but on exercise: the most relevant techniques are briefly explained and the majority of the time is devoted to exercising the techniques through one major case study, the Euro Wonderland case. Additional learning experiences come from the fact that one fully integrated case study is used to clearly demonstrate how techniques and deliverables are interdependent from a content and timing perspective (e.g., how work breakdown relates to a task list and how to get from a task list to an initial network schedule).
In these two days, the key PMBOK® Guide knowledge areas (quality management and procurement are only briefly handled because they part the subject of other courses) are presented through in at least one of the game scenarios. Obviously, that's a lot on their plate, but due to the fact that we use gamification principles, participants are really motivated to work/play around the clock, extending the normal 9 to 5 training hours. Actually, the game and the learning continues all night long (this is an off-site course with overnight stay) with an exercise in managing a team, communication management, change management when they are actually riding an antique train and run out of budget. This is just one example where life action is included to increase the learning experience.
The Evaluation of a Gamified Project Management Course
The return on investment (ROI) for training is often questioned. Although organizations spend millions of dollars on training and development, the effectiveness of training is often very low. The Kirkpatrick model evaluates the effectiveness of training by measuring changes in behavior that occurs as a result of training programs on four levels (HP, Measuring ROI of training, 2004, p. 7; Mindtools, paragraph 4-11).
- Level 1(“Reaction”): Did they like it?
- Level 2 (“Learning”): Did they learn?
- Level 3 (“Behavior”): Do they use it?
- Level 4 (“Result”): Did it impact the bottom line, what's the ROI?
Let us use this model for measuring the added value of using gamification elements to basic project management training:
Level 1 (Reaction): Did They Like It?
The offsite venue, the entertainment park context, the game design, and also the fact that teams need to compete against each other but also work together at different intervals are rated very high. These high satisfaction rates for the gamified course are a constant factor over all sessions since the start of the gamified course. Participants appreciate the extra investment (mainly in preparation time), the collaboration/exploration model and it provides an opportunity for team members to get to known each other better on an extra-professional level. This approach leads to more understanding (e.g., for differences in personal styles) and improved collaboration when they return to their actual project environments.
Level 1 is a basic but important level of evaluation, since it improves employee retention. Positive feelings about the course are also a prerequisite for actual learning during the course. The positive feelings about the course open up the person's mind, making learning possible.
On the level of retention, almost all past participants indicated that the gamified project management course increased their motivation to stay longer (and work harder!) for the company. Some employees that not (yet) have been enrolled, indicate that the gamified course is high on their wish list and an incentive for them to stay with the company.
On the first level of the Kirkpatrick model, we can conclude that this gamified course leads to increased employee satisfaction, returning an above market average employee retention figures for our company and supporting the company in remaining a “best place to work for” (“Best Workplaces is an international contest from the Great Place to Work Institute organized in Belgium in collaboration with the Vlerick Business School. CTG has been nominated in the Top 10 best place to work, 6 times in the last 11 years, with a +85% score for training and development in 2013).
Level 2 (Learning): Did They Learn?
The actual accrual of knowledge is measured right after the training course through a gamified quiz in which participants need to respond to game scenarios using the knowledge acquired in the previous days. The main gamification design elements used for this quiz are bonus and penalty points and the leader board mechanism…. The quiz is played on an actual mountain forcing the participants to go up and down the mountain depending on the answers they provide and depending on their confidence level level (e.g., how confident am I that I will have the next question correct providing only the knowledge area).
The participants are incentivized to collaborate (intra-group) and compete (inter-group) with a goal to get to the actual top of the mountain as fast as possible. What we found was that participants got higher results on the course end quiz organized at the end of the two-day course compared similar end tests in more traditional (“ex cathedra”) class settings.
Using game dynamics such as bonus and penalty points and leader boards incentives the participants to capture more knowledge per time unit, but also keeps them focused on learning for a longer time. The increased focus and attention span is an important side effect that is also noticeable in playing actual (video) games (Zichermann, 2013).
An additional positive learning effect comes from the fact that a game context (building an entertainment park) is used that has completely nothing to do with any of the services/projects the participants are working in (most of them are working in the IT services sector, some in Human Resources). This deliberate choice is made to allow the participants to focus on the value of the techniques rather than discussing in why for their specific technology or their specific customer environment this technique is not feasible. The transfer of knowledge to their actual project setting is done at the end of the course, when they have fully exercised all the techniques and at a time when they are able to evaluate the full integrated chain of techniques used. Although I advise to use a game context that is not theirs in reality, I also advise to use a game context that they can relate to, which is easy to understand for all participants. The simplified context of the entertainment park seemed to be the best choice over time since all participants know it and also because it's fun, which is also an important aspect of gamification. The higher degree of reality, the better the motivation, hence the more effective the course is.
On level 2 of the Kirkpatrick evaluation model, gamification makes people acquire more knowledge, both from a quality and quantity point of view.
Level 3 (Behavior): Do They Use It?
Where level 2 focuses on the acquisition of knowledge, level 3 is all about applying that knowledge in a real situation. This is measured by the changes in behavior. A best practice would be to measure the long-term behavioral change and therefor I would advise to measure level 3, only six months after the course. If people still use the technique after six months, one can consider this as a lasting effect of the training course. What we noticed at CTG is that project managers who followed the gamified project management course were more eager to use the techniques that were simulated, more than their colleagues who were already familiar with or experienced in the same techniques and did not follow the gamified project management training (for example, because they followed a basic training course with their former employer, or because they were more experienced to senior project managers). The techniques that were evaluated were all part of the CTG project management methodology, although not all of them were mandatory techniques.
The conclusion that can be derived from this is that not just explaining, but really exercising a technique through “safe” but similar circumstances increased the spontaneous use of those techniques. Providing a realistic but harmless context (aka “sandbox”) in which errors can be made and corrected, is a key element of a serious game.
This leads to the conclusion that gamified courses support a higher actual usage of knowledge acquired.
Level 4 (Result): Did It Impact the Bottom Line? What's the Return on Investment?
Gamifying the project management course had a direct impact on the company's bottom line on two levels.
First, the gamified version of the project management course was clearly cheaper (and rated better) than the classical ex-cathedra version. The additional costs for renting a house and the cost of gaming material (mainly paper, flipcharts, Post-it notes) are still lower than sending the same group to an external project management course of similar duration with overnight stays in hotel rooms and less fun.
Second, since more project managers could be trained for the same budget and since from a quality perspective the techniques were better understood (level 1 evaluation) and more used (level 2 evaluation), this leads to the effect that our actual IT projects for customers are being better managed. Over the years we clearly noticed: fewer projects that went over budget and, in case of an overrun, that on average budget overruns tend to get smaller and are detected earlier. In the knowledge area risk management for example, we notice a better application of risk planning, risk response execution, and contingency budget planning. The two effects mentioned above are the direct things impacting our bottom line. The indirect effects should not be underestimated: increased employee satisfaction, more collaboration on the actual projects, decreasing employee turnover…. The indirect gains are harder to measure but can by far outrun the direct benefits from gamification.
Conclusion
The Euro Wonderland case of CTG clearly indicates the positive return on investment for gamification in the area of project management education. With gamification becoming more well-known and successful in the next years, companies will be using it more as a means to increase the effectiveness of their training programs. But not only for their training programs, as pointed out in this paper, the gamification of the educational part of project management actually leads to better managed projects in the real business environment.