ABSTRACT
Simulation-Based Training (SBT) is considered a valuable methodology for project management education. This paper aims to identify its boundaries of effectiveness and investigating circumstances under which SBT is most effective. Three studies have been conducted with area experts and graduate management students in the United States and Australia, using a pre/post design, for various trainee individual differences. Results suggest that (1) SBT improves trainees’ attitudes toward the advantages of SBT as a training tool and enhances knowledge only when the gap between the trainees’ prior knowledge and the challenges presented in the simulation is at certain levels, and (2) SBT energizes learning only when trainees succeed in the simulation. Results of these studies suggest that SBT designers should advise lecturers of the required prior knowledge level of trainees and integrate various levels of difficulty into the design. Lecturers should ensure that students possess the required theoretical knowledge before the SBT session. Additionally, project management lecturers should not treat SBT as a standalone teaching technique and ensure that students possess the required theoretical knowledge before the SBT session, because it is most effective when supplementing it with other teaching methods.
Keywords: education; simulation-based training; learning outcomes; individual differences
INTRODUCTION
At the end of a Simulation-Based Training (SBT) session, three project management students were discussing what they had learned from the session and how they felt about the learning experience. David said “I feel so frustrated and I don't even understand what we were doing during the session.” Maggie quickly replied: “Really? I am so excited about the simulation game we just played and I have learned so many new concepts. For example, I now know how to better allocate resources for a new project. I think we should have this kind of simulation session more often in the future.” Bob disagreed: “Um, I don't know. Even though I do find it to be an interesting learning experience, I don't feel I have learned something new from this session.” Why do these three management students gain different results and feel differently about the same simulation session? The answer is simple: because “one size does not fit all.” SBT can be an effective teaching tool for some students, but may not work for others. Then, the next questions that arise are: (1) What are the SBT boundaries of effectiveness? (2) How should we design SBT tools, taking into account the boundaries of effectiveness? And (3), how should we use SBT for management education, taking into account the boundaries of effectiveness?
As project complexity increases, today's project managers need to possess specialized skills and knowledge as well as the abilities to adapt them to specific circumstances. Project management education plays a critical role in helping project managers acquire these competences, and accordingly, much effort has been expended on exploring innovative ways to enhance the quality of project management education. SBT is one of the many techniques that have been considered of great potential for effective management education (Bell et al., 2008; Salas et al., 2009). This paper uses the SBT definition provided by Salas et al. (2009): “a synthetic practice environment that is created in order to impart the competencies (e.g., attitudes, concepts, knowledge, rules, or skills) that will improve a trainee's performance.” This definition refers broadly to all types of computer-based simulations that involve the creation of synthetic learning environments.
In past decades, SBT has been increasingly used in management training in both academic and industry communities (Bell et al., 2008; Salas et al., 2009). For instance, Faria (1987; 1998) found that in 10 years, the increased use of SBT in management education from 18% to 44% was higher than in other business disciplines, such as marketing, accounting, and finance. In practice, 75% of U.S. organizations with more than 1,000 employees use SBT in business training (Faria & Nulsen, 1996). In accordance with this trend, many academic studies have been conducted to develop specific SBT systems (Draijer & Schenk, 2004; Wiendahl et al. 1995), examine the effectiveness of SBT in enhancing learning outcomes in comparison with other teaching techniques (Mitchell, 2004; Romme, 2004, Cannon-Bowers & Bowers, 2009), and develop a set of guidelines for effective use of SBT (Bell et al., 2008; Salas et al., 2009).
Although prior studies have enhanced our knowledge about SBT, they tend to focus mostly on success stories and seldom tell us under what circumstances SBT fails. More specifically, there has been a growing recognition that trainee individual differences (e.g., ability, prior knowledge, experience, and personality) can affect SBT effectiveness and that a “one size fits all” approach can result in ineffective training (Bell et al., 2008). Unfortunately, very little research has been conducted to examine how individual differences influence the overall effectiveness of SBT. As a result, it is difficult to determine for which group of trainees SBT is most effective, and how to design and use SBT to maximize learning enhancement for various groups of trainees. Our aims in this paper are therefore threefold: (1) to examine the advantages of SBT as a training tool in project management education; (2) to understand how individual differences may affect the effectiveness of SBT in project management education; and (3) to provide guidance on how to effectively design and use SBT to enhance the learning outcomes of project management students.
In order to achieve the designated research objectives, three studies were conducted in diverse project management classes using a simulator tool, called the Project Team Builder (PTB)—a training aid designed to facilitate the training of project management in a dynamic, stochastic environment (Shtub, 2011). In the remaining parts of this paper, we first review the literature on SBT in general and in the project management discipline specifically. Next, we present the theoretical frameworks and the proposed hypotheses. We then continue to describe the three studies conducted, present study results, and discuss research findings. Finally, we provide some guidance regarding the future design and utilization of SBT in project management education.
LITERATURE REVIEW
This section reviews the literature on SBT in general and in the project management discipline.
Literature on Simulation-Based Training (SBT)
SBT is considered an effective and dynamic training tool (Zantow et al., 2005), especially for the virtual generation of students (Proserpio & Gioia, 2007). Over the past decades, considerable effort has been devoted to studying issues in relation to SBT. For the sake of clarity, these prior studies are loosely grouped in this review into three categories: (1) introduction of a specific SBT system; (2) examination of the effectiveness of SBT; and (3) discussion on the advantages of SBT and guidelines for its utilization.
Studies in the first group usually focus on the introduction of a specific SBT system, describing in detail its underlying assumptions and technological features, and its applicability in a particular content (e.g., medical, military, or pilot trainings). For example, Draijer and Schenk (2004) presented the use of an SBT tool called “SAP R/3” to support learning of business processes and process integration. They argued that this tool is effective in enabling problem-based learning and practicing business simulation. The main advantage of this type of SBT studies is that they provide detailed information (e.g., its advantages, functionalities, and limitations) about the SBT presented, which is useful for educators who intend to use the particular SBT tool. Such studies, however, provide little insight into how effective the SBT tool is as compared with other training techniques.
Studies in the second group overcome the above limitation by comparing the relative effectiveness of SBT with other teaching techniques. For example, based on quasi-experimental studies Wolfe (1997) found that simulations produce better learning outcomes than business case studies. Cannon-Bowers and Bowers (2009) concluded that simulations are effective in a variety of contexts such as training of pilots, clinicians, military personnel, firefighters, and survey interviewers. Several studies have also found that learners generally report positive attitudes (e.g., satisfaction) toward the use of simulations in training and education (Mitchell, 2004; Romme, 2004). Despite the above supporting evidence, the effectiveness of SBT was not confirmed in some studies (Cameron & Dwyer, 2005). Some researchers (Tonks & Armitage 1997; Gosen & Washbush, 2004) further argued that there is still a lack of rigorously conducted studies on the effectiveness of SBT. For example, Keys and Wolfe (1990, p. 311) criticized studies on the teaching power of business games by stating that many are rested on “anecdotal material or inadequate or poorly implemented research designs.”
Studies in the third group frequently discuss the general advantages of SBT as a training tool and provide guidelines for its future utilization. For example, Salas et al. (2009) suggested that SBT offers several advantages such as imparting complex applied competencies, and providing a realistic risk-free learning environment. The authors also proposed a list of guidelines for effective SBT implementation in management education. Even though such studies provide an overview on why SBT is an effective training tool and how it can be more effectively utilized, they tend to suffer from a lack of empirical validation.
SBT in Project Management Training
A project is “a temporary organization and process set up to achieve a specified goal under the constraints of time, budget, and other resources” (Shenhar & Dvir, 2007). As a result, project management aims at delivering an output on time, on budget, and according to the specifications (Zwikael & Sadeh, 2007). This is a very challenging organizational task for several reasons. First, projects are usually implemented in a complex and dynamic environment, where great uncertainty exists (Zwikael & Globerson, 2006). In addition, difficult trade-off decisions have to be made between the three project management attributes; namely, time, cost and specifications. To tackle these challenges, the management discipline should support graduates in developing relevant knowledge, as well as a wide range of (professional and personal) skills. For example, a project manager must be able to comprehend complex patterns of information, diagnose and understand the environment, manage uncertainty, monitor project cash flow, and perform complex trade-off analyses and act upon the results of those analyses (Zwikael & Ahn, 2011). Preparing project managers for these tasks requires training tools that allow them to deal with uncertainty and explore different possibilities (Thomas & Mengel, 2008).
Due to the importance of projects in achieving organizational strategies, interest in project management is growing (Thomas & Mengel, 2008); and project management has been included as a discipline in management education. The possibility and effectiveness of utilizing SBT in project management education have been explored by several researchers. These prior studies generally fall into two broad categories. One category focuses on the development of SBT tools for the use in a specific project management domain (e.g., engineering project management or software project management). Some examples of such studies are Davidovitch et al. (2006, 2008, 2009, 2010) and Nembhard et al. (2009). The second category examines the effectiveness of SBT in project management education as compared with other traditional teaching tools, such as in Pfahl et al. (2004).
This short literature review suggests that the advantages of SBT as a training tool are well perceived and, accordingly, a variety of SBT tools have been developed. The effectiveness of SBT as compared with that of other training techniques is however not absolute. In the studies where the advantages of SBT have not been supported, the personal characteristics of learners were found to be a moderating cause. Such findings are consistent with the argument that “one size does not fit all” (Shenhar, 2001). In other words, individual differences must be taken into consideration for SBT to be effective. Unfortunately, the majority of prior studies on SBT have been dominated by the “one size fits all” approach (Bell et al., 2008). As a result, the influence of individual differences on the design and utilization of SBT is still poorly understood. This paper is an attempt to study the effectiveness of SBT and its boundaries in project management education.
THE THEORETICAL FRAMEWORKS AND HYPOTHESES
SBT Advantages for Management Education
The basic claim that triggers this research is that SBT is an effective management training method in seven various manners (Salas et al., 2009): (1) SBT is superior to other training strategies for imparting complex applied competencies; (2) SBT provides a more complex and realistic learning environment than other training strategies; (3) SBT provides a relatively risk-free environment for learning and experimentation; (4) SBT is an ideal method for training infrequently engaged but critical skills; (5) SBT is usually simple to learn and operate; (6) SBT is a form of learner controlled learning: and (7) SBT is inherently more engaging than other training methods. This research first aims to examine the influence of trainees’ exposure to SBT on their perceptions of these proposed advantages. Prior studies (Mitchell, 2004; Romme, 2004) suggested that SBT can encourage a more positive attitude toward learning. Accordingly, we argue that SBT can improve trainees’ attitudes toward the perceived SBT advantages and propose that:
H1a: Exposure to SBT encourages a more positive attitude toward its advantages.
Prior studies have suggested that individual differences can influence the effectiveness of SBT (Bell et al., 2008). The influence of individual differences on the effectiveness of learner-controlled training techniques (e.g., SBT) is especially significant (Brown et al., 2002). For this reason, we are interested in testing whether such individual differences also affect trainees’ attitudes toward the advantages of SBT. Trainees’ individual differences can be in different areas, such as their education backgrounds, personalities, skills, and prior knowledge levels. Among these, the trainees’ prior knowledge on the subject has been considered as one of the most important determinants of learning outcomes by many researchers and educational psychologists (Hilgard & Bower, 1975; Haugeland, 1985). This research therefore focuses on investigating whether trainees’ prior knowledge levels play a role in the changes of their attitudes toward the advantages of SBT. The hypothesis we propose:
H1b: Level of prior knowledge moderates the relationship between SBT and trainees’ attitudes toward SBT advantages.
The theoretical framework for the above research objectives is illustrated in Figure 1.
Figure 1: Theoretical framework for trainees’ attitudes toward SBT advantages.
Effectiveness of SBT on Knowledge Enhancement
One specific SBT advantage is that it enhances trainees’ knowledge (Wolfe, 1997; Washbush & Gosen, 2001). Hence, the second aim of this research is to examine the effectiveness of SBT with regard to this goal and investigate the circumstances under which SBT is effective. The first hypothesis we propose for this objective is:
H2a: SBT enhances trainees’ knowledge.
The relationship between a trainee's prior knowledge level and learning outcomes has been well-established in the literature (Dochy et al., 1999). For example, Haugeland (1985) argued that the trainees who have sufficient prior knowledge learn more than those who do not and suggested that the difficulty level of a learning material needs to be understood, taking into consideration both the way in which the material is presented and the knowledge level of the audience for which the material is intended.
Some researchers (DeRouin et al., 2004; Lee et al., 2006) argued that SBT is most effective when a trainee's abilities, prior experience, and knowledge in the content domain are high. It is argued that in this case trainees can focus on the learning activities (such as effective learning decisions) rather than on the acquisition of important knowledge and skills. Interestingly, however, it has been found in some studies that students with high prior knowledge do not necessarily gain the most benefit from training. For example, Seufert (2003) found that the learning outcomes of students with high prior knowledge level are not enhanced in a multimedia learning environment, whether the directive or non-directive help is offered or not. The author argued that this may be due to an illusion of knowing: students with high prior knowledge may overestimate their own abilities and hence underestimate the benefits from the training process.
Based on these competing propositions, we suggest that SBT enhances a trainee's knowledge only when the gap between the prior knowledge and the challenges presented by the simulator is at the right level. In other words, SBT enhances knowledge only when the a priori knowledge needed to run the simulation is available. If a trainee does not have the required prior knowledge in the discipline area, he or she may not be able to use a simulator, and hence will not learn from the exercise. If, on the other hand, a trainee has too much prior knowledge, the exercise may get too easy, and an SBT may be found an ineffective teaching tool. Based on this argument, we propose:
H2b: Level of prior knowledge moderates the relationship between SBT and knowledge enhancement.
Circumstances under which SBT Energizes the Learning Process
A number of studies (Mitchell, 2004; Romme, 2004) have reported that individuals generally have positive reactions (such as satisfaction) to the use of SBT in training. For example, students were found to have more positive attitudes toward learning from business games than from other teaching approaches (Malik & Howard, 1996). Salas et al. (2009) suggested that the inherent entertainment characteristics of SBT (e.g., simulation and game-based) make it more likely to engage students in the learning process than other teaching methods, such as lectures. Such engagement is expected to increase a student's learning interest and motivation, leading to long-term effective training (Mathieu & Martineau, 1997). In terms of management training, the capability of SBT to increase interest, involvement, and enthusiasm of students in class has been confirmed by researchers such as Keys and Wolfe (1990) and Grishop (1987). Based on these arguments, we propose:
H3a: SBT energizes the learning process.
Despite the above-mentioned supportive evidence, some researchers (Remus & Jenner, 1981; Faria, 1987) argued that the enjoyment of using SBT depends on the degree of success achieved. From our own long-term experience with PTB, we found this argument convincing. Students’ learning processes are mostly energized when they get positive feedback from the simulation. In other words, students who do not succeed in the simulation may feel frustrated, and less energized with their learning processes. For this reason, we propose that the learning process is energized only when students succeed in simulation, that is, the student is able to complete all required tasks with a level of achievement above the expected threshold.
H3b: Success in a simulation moderates the relationship between SBT and the level of the learning process is energized.
The theoretical framework for examining the effectiveness of SBT in enhancing knowledge and energizing learning is shown in Figure 2.
Figure 2: Theoretical framework for SBT effectiveness in enhancing knowledge and energizing learning.
METHODS AND SAMPLES
This section describes the simulation tool used, and the three studies conducted to test the proposed hypotheses.
The SBT Tool: Project Team Builder (PTB)
To test the effectiveness of SBT for teaching project management we used a research tool called Project Team Builder (PTB)—a training aid designed to facilitate the training of project management in a dynamic, stochastic environment (Shtub, 2011). PTB has been used as a teaching and research tool in project management education for several years (Davidovitch et al., 2006, 2009, 2010; Nembhard et al., 2009).
In PTB, several built-in project scenarios with different difficulty levels are provided. For each scenario, three types of data are given, including general information, task information (e.g., the distribution of the duration of each task, the predecessors of each task), and resource information (e.g., types of resources and their availability and the project cost). Each project scenario has a due date and penalty may be applied in the case of late completion, while a financial incentive may be added in the case of an early completion. A trainee's goal is to plan and execute a selected project scenario with the minimum cost, in other words, to maximize the cash position at the end of each project scenario. Trainees could increase the cash by proper scheduling of activities, which reduces penalties for late delivery, employee hiring and firing costs, subcontractor cost, and rushed mode of work that involves overtime.
In the PTB simulation process, a trainee first chooses a project scenario and then defines project tasks (e.g., setting the starting date, splitting the task). He or she then needs to plan project tasks to meet resource constraints. In some scenarios, the trainee is also allowed to choose different task execution modes (e.g., lower cost with longer duration or higher cost with shorter duration). In simple (deterministic) project scenarios, the execution of the project is performed exactly according to the plan. However, in the advanced (stochastic) scenarios, some uncertainty exists with task actual duration and cost, and resource availability. In this case, further planning and adjustments of project tasks may be necessary. The simulation ends when all project tasks are completed.
PTB is controlled by a simple user interface and no knowledge of simulation or simulation languages is required. Figure 3 presents two screen shots from PTB.
Figure 3: A network diagram and resource allocation view of a project scenario in PTB.
Studies and Participants
We used PTB as a research tool in three studies conducted at leading universities in the United States and Australia. Studies varied in their objectives, levels of prior knowledge of participants (faculty and students with mixed prior knowledge levels on project management), and countries. This design is considered suitable for testing our hypotheses, which require heterogeneous groups of participants, especially in terms of their prior knowledge levels on project management. These studies aimed to validate the advantages of SBT and identify the circumstances under which SBT enhances trainees’ knowledge and energizes the learning process. The three studies are summarized in Table 1 and described in the following sections.
| Study 1 | Study 2 | Study 3 | |
| Objective | Test hypotheses on SBT advantages for project management education | Test the effectiveness of SBT on knowledge enhancement and learning energization; and to identify the circumstances under which it is more effective | Combine the objectives of both previous studies |
| University | Pennsylvania State University | The Australian National University | The Australian National University |
| Program | School seminar for faculty | MBA (project management course as an elective) | MBA (project management course as an elective) |
| Number of participants | 14 | 32 | 23 |
| Number of groups | 1 | 2 | 2 |
| Duration of each session (hours) | 3 | 5 | 3 |
| Dates | September 2010 | July–August 2010 | April–May 2011 |
Table 1 – Descriptions of the three studies.
Study 1
The first study was conducted through a research seminar involving 14 faculty and staff members from Pennsylvania State University. The objective of this study was to examine the changes in trainees’ attitudes toward SBT advantages for project management education. A pretest and post-test procedure was adopted (Zikmund et al., 2010). At the beginning of the seminar, participants were first asked to report on a 7-point Likert scale their levels of agreement with each of the seven SBT advantages as a teaching tool (Salas et al., 2009). An example advantage is “SBT (Simulation-Based Training) is superior to other training strategies for imparting complex applied competencies.”
Then, a hands-on workshop using PTB was delivered, in which participants learned the principles of the simulation and run several scenarios using the tool. At the end of the seminar, participants were asked to answer the same set of questions included in the first questionnaire again. The pre- and post-session questionnaires were matched based on the personal codes provided by participants.
Study 2
The objectives of Study 2 were to test the effectiveness of SBT in enhancing knowledge and energizing learning process, and to identify the circumstances under which it is more effective. For these purposes, data were collected in a class of 32 postgraduate management students at the Australian National University (ANU) as part of an elective course on project management. Sixty-eight percent of participants had work experience of three or more years and 50% with project management experience. Seventy-three percent of participants were male and 27% at the age of 30 or under.
For some students, this was the only project management course in their academic program; others had taken previous courses in this area, whereas some practice project management daily in the workplace. The mixed backgrounds of the students and the diversity of courses they had taken in the area of project management made it possible to test our hypotheses regarding the entry conditions required for an effective SBT session.
Similar to Study 1, we used two matched questionnaires. The first was administrated at the beginning of the SBT session, and the second at its end. The first questionnaire included a set of knowledge questions on project planning, monitoring and control, and a question related to students’ levels of agreement with the statement that SBT energized their learning adapted from Grishop (1987). The simulation session started with a short introduction to PTB and an illustration with a simple project scenario. Following the introduction, students were asked to run easy and intermediate project scenarios and then progress to the advanced one. The characteristics of the three scenarios are summarized in Table 2. Students were asked to report the cash position at the end of each simulation they had completed in the advanced scenario.
| Scenario 1 Easy | Scenario 2 Intermediate | Scenario 3 Advanced | |
| Number of activities | 6 | 6 | 12 |
| Activity duration | Deterministic | Stochastic | Stochastic |
| Resource constrained | No | Yes | Yes |
| Cash constrained | No | No | Yes |
Table 2: Characteristics of the three scenarios used in the controlled studies.
The whole session took five hours, with an additional debriefing session on the following week. At the end of the simulation session, participants were asked to complete a second questionnaire, which repeated the questions included in the first questionnaire, but also asked for demographic information, such as age, gender, work experience, and project management experience. All questionnaires were anonymous and the results of the simulation runs had no effect on students’ grades in the course. For data analysis, the pre- and post-session questionnaires each student had completed and their simulation result sheet were matched using an individual code of the student’ s choice.
Study 3
Study 3 integrated the objectives of both previous studies. However, unlike Study 1, it involved postgraduate students and differed from Study 2 in that it also included questions about SBT advantages as a teaching tool for management education. Using the same approach described in Study 2, participants were asked to complete knowledge tests and report their agreements on SBT advantages (Salas et al., 2009) and levels of energized learning (Grishop, 1987) before and after the PTB session. The before and after questionnaires and the form with their reported results of simulations were matched using a personal code, as in previous studies. Sixty-two percent of participants had work experience of three or more years, with 37% with project management experience. Fifty percent of participants were male and 77% at the age of 30 or under. The 23 students were divided into two groups, and each session took three hours.
MEASURES
We used multiple measures in these studies:
SBT
SBT was measured as a binary variable. Lack of SBT experience refers to students prior to the training session, assuming they have not had experience with the particular SBT being used. Existence of SBT experience refers to the same group of students following the simulation session.
SBT Advantages
Seven potential advantages of SBT for management education as suggested by Salas et al. (2009) were tested. A list of these seven advantages can be found in preceding parts of this paper. A seven-point Likert scale, ranging from “strongly agree” to “strongly disagree,” was used. This scale was used in studies 1 and 3, before and after the simulation sessions. Cronbach alpha results in Study 1 were 0.78 and 0.80, whereas in Study 3 they were 0.82 and 0.86. These values satisfy the minimum requirement for scale reliability (Garmezy et al., 1967).
Knowledge Enhancement
Knowledge enhancement is determined based on the differences between a student's scores in the knowledge tests conducted prior and following the SBT session.
Prior Knowledge—In the beginning of the simulation session, we asked students a set of knowledge questions on aspects of project management relevant to the simulation, which include project planning, monitoring, and control. For example, one question asked students to name approaches to reduce a project's level of uncertainty. Students were marked with one point for each correct answer they provided. The score for this measure was the overall mark a student received for all knowledge questions.
Knowledge Following SBT—Following the simulation session, students were asked to answer the same set of knowledge questions. The results were marked using the identical scoring approach as described above.
Energizing the Learning Process
Before and after the session, students were also asked to mark the extent to which they agree with the following statement: “SBT has energized my learning process” using a seven-point Likert scale. This question, adapted from Grishop (1987), was used to allow an analysis of the relationship between SBT and the level by which the learning process is energized.
Limitations
Limitations of this research include the limited focus on knowledge enhancement and learning energization, which are only two measures of learning outcomes, and the use of a single item index to measure the energized learning process. In addition, this research focuses only on trainees’ differences in their prior knowledge levels. Further studies are needed to examine how other individual differences such as personality or skills may affect the effectiveness of SBT. Finally, the studies were limited to a relatively small number of participants and the use of a single SBT tool.
RESULTS
This section presents the results of each of the three studies separately. Results were controlled for age, gender, position type, work experience, and project management experience of participants, as well as industry and sizes of the organizations they work in.
Study 1
The levels of agreement regarding SBT advantages for project management education before and after the simulation session were compared using a paired sample analysis. Table 3 presents the mean and standard deviations, as well as t value to test the significance of the difference. Results show that the level of agreement with SBT advantages was significantly higher after the simulation session than before it began for all seven items.
| Study 1 | Study 3 | |||||
| SBT Advantages for Project Management Education | Mean (STD) before simulation | Mean (STD) after simulation | t value (two tailed) | Mean (STD) before simulation | Mean (STD) after simulation | t value (two tailed) |
| SBT is superior to other training strategies for imparting complex applied competencies | 4.43 (1.28) | 5.57 (0.85) | -3.040 ** | 4.77 (0.97) | 5.09 (1.34) | -1.231 ns |
| SBT provides a more complex and realistic learning environment than other training strategies | 5.14 (1.03) | 6.07 (0.92) | -2.253 * | 4.95 (1.05) | 5.27 (1.08) | -1.275 ns |
| SBT provides a relatively risk-free environment for learning and experimentation | 5.36 (1.28) | 6.57 (0.51) | -4.323 ** | 5.27 (1.28) | 5.50 (1.41) | -1.045 ns |
| SBT is an ideal method for training infrequently engaged but critical skills | 5.21 (0.98) | 6.43 (0.65) | -3.634 ** | 4.81 (1.17) | 5.29 (1.35) | -1.644 ns |
| SBT is usually simple to learn and operate | 4.14 (1.61) | 6.14 (0.95) | -4.770 *** | 4.40 (1.23) | 4.30 (1.78) | 0.346 ns |
| SBT is a form of learner controlled learning | 5.07 (1.00) | 6.29 (0.61) | -5.090 *** | 4.75 (0.97) | 5.20 (1.20) | -0.917 ns |
| SBT is inherently more engaging than other training methods | 5.21 (1.12) | 6.14 (0.95) | -2.183 * | 4.45 (1.39) | 5.00 (1.56) | -0.718 ns |
Table 3: SBT advantages for learning enhancement.
*p<0.05; **p<0.01; ***p<0.001; ns: non-significance
Study 2
A paired samples analysis was conducted to compare a trainee's knowledge level before and after the simulation session. We first analyzed data from all participants together, then divided them into three sub-groups, low, medium, and high prior knowledge levels, based on their median scores on knowledge questions prior to the PTB session. Table 4 presents the results of this analysis. It should be noted that students with low prior knowledge level are not included in Table 4 because the majority of them did not complete the simulation nor fill in the post-test knowledge questionnaires.
| Study 2 | Study 3 | |||||
| Prior knowledge | All | Medium | High | All | Medium | High |
| N | 21 | 15 | 6 | 23 | 8 | 15 |
| Mean knowledge level before simulation (STD) | 3.68 (1.73) | 2.80 (1.15) | 5.83 (0.98) | 4.09 (1.12) | 2.88 (0.58) | 4.73 (0.73) |
| Mean knowledge level after simulation (STD) | 4.91 (1.37) | 4.53 (1.41) | 5.67 (1.03) | 4.00 (1.12) | 3.69 (0.96) | 4.17 (1.19) |
| t value | -3.462 | -4.840 | 0.237 | 0.364 | -2.600 | 2.241 |
| Significance value (2-tailed) | 0.002** | 0.000*** | 0.822ns | 0.719ns | 0.035* | 0.042* |
Table 4: Impact of prior knowledge level on knowledge enhancement.
*p<0.05; **p<0.01; ***p<0.001; ns: non-significance
As can be seen from Table 4, a significant level of knowledge enhancement was observed for the entire group of students. For students with a medium level of prior knowledge on the subject, their knowledge was significantly improved. However, for students with a high level of prior knowledge, there was a minor insignificant decline in their knowledge.
A paired sample mean comparison test was conducted to analyze students’ agreement with the statement that SBT energized their learning, based on their reports in the pre- and post-session questionnaires. Similar to the previous analysis process, we first analyzed students as a whole and then divided them into two sub-groups: succeed and failed in simulation. Success in simulation, as explained to all students in an introductory lecture, was measured by the amount of cash they had at the end of the simulation of an advanced project scenario. A simulation is considered as successful only when a positive cash position is reported. Twelve students who completed the advanced scenario without going into bankruptcy were placed in the succeed group, whereas students who were unsuccessful in completing the advanced scenario were positioned in the failed group. Results of a paired samples t-test are presented in Table 5.
| Study 2 | Study 3 | |||||
| Simulation result | All | Succeed | Failed | All | Succeed | Failed |
| N | 21 | 12 | 9 | 20 | 9 | 11 |
| Mean score before simulation (STD) | 5.45 (1.18) | 5.00 (1.13) | 5.89 (1.05) | 4.60 (1.14) | 4.22 (1.09) | 4.91 (1.14) |
| Mean score after simulation (STD) | 5.78 (0.82) | 5.58 (0.67) | 6.00 (1.00) | 4.95 (1.54) | 5.56 (1.42) | 4.45 (1.51) |
| t value | -1.884 | -0.250 | -0.430 | -1.000 | -3.578 | 1.047 |
| Significance value (2-tailed) | 0.073ns | 0.027* | 0.681ns | 0.330ns | 0.007** | 0.320ns |
Table 5: SBT energizes the learning process for various levels of success in the simulation.
*p<0.05; **p<0.01; ***p<0.001; ns: non-significance
Results show that students’ agreement with the statement that SBT energized their learning increased insignificantly after the simulation session. Significant results were found only for students who succeeded in the game, as they felt that the simulation session energized their learning process. Students who failed the game did not feel the simulation energized their learning process.
Study 3
As the objectives of Study 3 combine the objectives of Studies 1 and 2, we repeated the same analysis processes that have been used in the previous two studies. First, we analyzed the hypotheses regarding SBT advantages in the same way as in Study 1. Results in Table 3 show that unlike in Study 1, in Study 3, there was no significant improvement in students’ agreements with the seven SBT advantages after the PTB session. However, slight improvements of students’ attitudes toward six SBT advantages (except the one that suggests SBT is simple to learn and operate) are still observed. In order to explain these contradictory results, we have conducted a more detailed analysis of the results. As Study 1 contained only faculty with high prior knowledge level, we split the students of Study 3 into two groups based on their prior knowledge levels. One group contains data from students with a high level of prior knowledge and the other of students with a medium level of prior knowledge. The results are presented in Table 6.
| Level of Prior Knowledge | Medium | High | ||||
| SBT Advantages for Project Management Education | Mean (STD) before simulation | Mean (STD) after simulation | t value (two tailed) | Mean (STD) before simulation | Mean (STD) after simulation | t value (two tailed) |
| SBT is superior to other training strategies for imparting complex applied competencies | 4.75 (0.89) | 4.38 (1.41) | 0.664 ns | 4.79 (1.05) | 5.50 (1.16) | -3.680 ** |
| SBT provides a more complex and realistic learning environment than other training strategies | 4.75 (1.04) | 5.00 (1.07) | -0.798 ns | 5.07 (1.07) | 5.43 (1.09) | -1.000 ns |
| SBT provides a relatively risk-free environment for learning and experimentation | 5.25 (1.28) | 5.38 (1.77) | -0.424 ns | 5.29 (1.33) | 5.57 (1.22) | -0.939 ns |
| SBT is an ideal method for training infrequently engaged but critical skills | 4.29 (0.76) | 4.57 (1.72) | -0.505 ns | 5.07 (1.27) | 5.64 (1.01) | -1.665 ns |
| SBT is usually simple to learn and operate | 3.71 (0.49) | 3.14 (1.35) | 1.188 ns | 4.77 (1.36) | 4.92 (1.71) | -0.433 ns |
| SBT is a form of learner controlled learning | 4.43 (0.98) | 4.29 (1.25) | 0.420 ns | 4.92 (0.95) | 5.69 (0.86) | -2.739 * |
| SBT is inherently more engaging than other training methods | 4.57 (1.72) | 3.71 (1.60) | 2.521 * | 4.38 (1.26) | 5.69 (1.03) | -4.571 ** |
Table 6: SBT advantages for management education in Study 3 moderated by prior knowledge levels.
*p<0.05; **p<0.01; ***p<0.001; ns: non-significance
Results in Table 6 show that for students with a medium level of prior knowledge, there was no significant change in the level of their agreements with six SBT advantages after the simulation. With respect to the advantage of SBT in engaging students, a significant reduced level was recorded. On the other hand, for students with a high level of prior knowledge, there was a significant increase in their agreements with SBT advantages in imparting complex applied competencies, enabling learner controlled learning and engaging students. For the rest of the SBT advantages, even though improvements in participants’ attitudes toward SBT advantages are observed, the results are not significant.
Then, we also repeated the analysis conducted in Study 2 to examine the impact of a trainee's prior knowledge level on knowledge enhancement. The results are presented in Table 4. Strikingly, we found that there is a non-significant decline in students’ knowledge after the simulation. This decline in knowledge is significant for students with a high level of prior knowledge. For students with a medium level of prior knowledge, there was a significant increase in the level of knowledge after the simulation.
Finally, the analysis of level of energized learning from Study 2 was repeated and the results are presented in Table 5. The results in Study 3 are similar to those in Study 2. There is an insignificant increase in students’ agreement with the statement that SBT energizes their learning. However, only student who succeeded in the simulation felt significantly more energized in learning following the simulation session than it is before the simulation. On the other hand, even though the results were insignificant, students who failed to complete the simulation stated that their level of energization was reduced.
DISCUSSION
This section integrates results from the three studies to discuss the various hypotheses.
SBT Advantages for Learning Enhancement
The first research hypothesis suggested that SBT has advantages for learning enhancement (Salas et al., 2009). All seven potential advantages were confirmed in Study 1, which only involved faculty with a high level of prior knowledge on the subject. However, in Study 3, where a mixed group of students with different levels of prior knowledge were included, results were not significant when all observations were analyzed together. When the group was divided by the levels of prior knowledge, it was found that students with a medium level of prior knowledge, in general, did not find SBT effective as a teaching tool. Specifically, even though the results are insignificant, following the SBT session, students of this group reduced their level of agreement with the statement that: SBT is simple to learn and operate and is a form of learner controlled learning. They were also less agreeable with the arguments that SBT is superior to other training strategies in imparting complex applied competences and in engaging them. This result is understandable, because the materialization of these SBT advantages is highly dependent on a student's prior knowledge level. In other words, students should have sufficient prior knowledge to complete the exercises in the SBT secession, so that they feel it is learner-controlled, feel engaged in the process, and learn the knowledge, which the SBT intended to deliver. On the other hand, if students do not possess the sufficient prior knowledge, they will not be able to run the simulation and as a result may feel frustrated. This may explain why students’ level of agreement to SBT's advantage in engaging them significantly decreased after the SBT session. For students with a high level of prior knowledge, their levels of agreement to all SBT advantages increased after the simulation. Among which, significant results are found for SBT advantages in imparting complex applied competencies, enabling learner controlled learning and engaging students. Students of this group had sufficient knowledge to complete exercises in the SBT exercise and gain the required knowledge outcomes. Students were able to control their learning processes and hence felt more engaged.
The Impact of Prior Knowledge on SBT Knowledge Enhancement
H2a suggested that SBT enhances knowledge. This hypothesis is significantly confirmed in Study 2 but could not be supported in Study 3. Furthermore, it was found in Study 3 that SBT was not just unable to enhance knowledge but also could result in a decline in knowledge. Although this result did not reach statistical significance, probably due to the sample size, it implies that SBT may not be effective in enhancing knowledge under all circumstances. In other words, SBT has its boundaries.
Our H2b hypothesis suggested that a trainee's level of prior knowledge affects the effectiveness of SBT in enhancing knowledge. In Studies 2 and 3 we found that SBT enhances knowledge only for students with an appropriate level of prior knowledge—a level high enough to allow students to actively participate in the session, but lower than achievement of the learning outcomes prior to the SBT session. The level of knowledge of students with an already high level of prior knowledge was declined following the simulation. A graphical representation of these results for Studies 2 and 3 is shown in Figure 4.
Figure 4: The impact of prior knowledge level on knowledge enhancement.
The following paragraphs provide detailed discussions on the effectiveness of SBT in enhancing the knowledge of students with different prior knowledge levels.
Low Level of Prior Knowledge—Participants in both studies were all graduate students in management with some work experience. However, some of them had not learned project planning, monitoring, and control (the knowledge required for the simulation) in any course prior to the experiment. Students with a low level of prior knowledge quit the simulation with much frustration and did not complete the second questionnaire after the simulation. As no theoretical knowledge was provided in this session, some of the students included in this group mentioned in a focus group session one week after the simulation that a “theoretical session before the simulation run was needed” and that “the simulation can be a good addition to a lecture.” These statements are in line with Salas et al. (2009) who claimed that SBT is “an advantageous way to supplement current educational methods.”
Medium Level of Prior Knowledge—This group of students achieved the highest level of knowledge enhancement after the simulation session in both studies. They had the appropriate knowledge in project planning and control to run the simulation (e.g., knowledge of how to identify the critical path and its implications on project outcomes), and the hand-on game provided them with a better understanding of the theoretical knowledge they had gained from previous courses. This implies that SBT enhances knowledge only when the level of prior knowledge is appropriate to the simulation's level of difficulty, and when the students possess the right knowledge gap.
High Level of Prior Knowledge—Our analysis revealed that there is no knowledge enhancement for students with a high level of prior knowledge. In other words, students with “too much” knowledge did not gain new knowledge from the SBT session. Even more strikingly, a decline in knowledge was observed in both studies for this group of students. One possible reason for this result is that students with a high level of prior knowledge had already gained the new knowledge the SBT was intended to deliver before the session (from theoretical knowledge and real life work experience); hence, they may put little effort in answering the post-test knowledge questions, such as by not attempting to repeat all answers.
It is important to mention that even though SBT was found to be not effective in enhancing knowledge of students with high prior knowledge level, it may enhance these students’ learning on some other aspects. The knowledge tests used in our studies can test students’ explicit knowledge; yet, they do not allow us to capture students’ improvement in implicit knowledge. Bell et al. (2008) suggested that SBT may create knowledge that is more implicit than explicit because it promotes experiential and discovery learning. For example, Swaak and de Jone (2001) found that SBT has a positive impact on the implicit knowledge measures, but has no effect on the traditional explicit knowledge measures. Such argument was supported by the discussions with the students in a session one week after the simulation. For example, one student mentioned that even though PTB did not enhance his theoretical knowledge, it improved his understanding about the decision-making process in project management.
The relationship between trainees’ prior knowledge levels and the effectiveness of SBT in knowledge enhancement is conceptualized in Figure 5. In the figure, the right-hand side curve is developed based on our findings, which is that SBT is most effective in knowledge enhancement for students with a medium level of prior knowledge. Students with a high prior knowledge level do not gain much new knowledge through SBT. Their levels of knowledge may even decline after the SBT session. However, due to the limitation of the sample size, more studies are needed to examine this finding. As students with a low prior knowledge level did not report on their results, the left-hand side curve in Figure 5 is hypothesized based on student comments and the relevant literature. Further studies are needed to examine the effectiveness of SBT in enhancing knowledge of students with a low prior knowledge level.
Figure 5: A proposed relationship between prior knowledge levels and knowledge enhancement.
Circumstances under which SBT Energizes the Learning Process
H3a claimed that SBT energizes the learning process. This hypothesis is confirmed, though not significantly, in Studies 2 and 3. The other hypothesis, H3b, suggested that SBT energizes the learning process only when students succeeded in the game. This hypothesis is significantly confirmed in both Studies 2 and 3. Students who failed in the game felt less energized in the learning process. Even worse, students’ energy in learning the subject can be diminished when they fail the simulation. Figure 6 presents these results graphically.
Figure 6: The impact of simulation success on energized learning.
An immediate implication of this result is the need to design SBT, with the flexibility to introduce varying levels of difficulty. In this case, students can choose the level that best fits their current skills and knowledge level and progress to more difficult levels at their own pace to achieve target learning outcomes. A scenario that is too difficult (or too easy) may cause frustration (or boredom) with the students, and hence will not energize their learning processes, nor enhance learning.
CONCLUSIONS
Simulation-Based Teaching (SBT) is described in the literature as a promising tool for creating a realistic, experimental learning environment (Bell et al., 2008). However, previous studies have shown mixed results regarding the effectiveness of SBT as a teaching tool, and individual differences among trainees were found to be critical contributors. Results from three studies conducted at Pennsylvania State University and the Australian National University described in this paper confirm the advantages of SBT for learning enhancement in management education (Salas et al., 2009), although they also highlight the fact that, as with other teaching approaches, SBT has its boundaries of effectiveness.
Because one size does not fit all (Shenhar, 2001), this paper investigated the circumstances under which an SBT tool, called PTB, is effective in studies involving a diverse group of management students and faculty. It was found that only when prior knowledge in the relevant area was at the correct level, trainees’ knowledge was enhanced through SBT. When students did not have the necessary theoretical knowledge, they had difficulties in running the simulation, and hence knowledge enhancement was limited. On the other hand, when students had possessed the intended knowledge outcomes before the simulation session, SBT did not enhance knowledge further. In other words, if the knowledge gap is too wide (i.e., students do not have the theoretical foundations for the simulation exercise) or too narrow (i.e., students already achieved all intended knowledge outcomes prior to the session) SBT is ineffective. We also found that SBT energizes learning only when trainees succeed in the game. This paper advances our understanding on the impact of individual differences in prior knowledge levels and in performances in simulation on the effectiveness of SBT. Such information is useful in designing courses using SBT in management education.
The results of these studies suggest several practical implications. First, designers of simulators should take into account prior knowledge levels of trainees for whom the simulation is designed in order to avoid common traps in simulation—boredom on one hand and incompetency to complete designed tasks on the other hand. A possible solution to avoiding this trap is to integrate varying levels of difficulty into the design of a SBT. For example, this solution was adopted in the design of PTB the Project Team Builder. PTB has two modules—a scenario generator and a simulator engine. The teacher can generate scenarios at a level of difficulty that fits students’ prior knowledge and the desired learning outcomes, thus not only minding the gap between the challenge presented by the scenario and the prior knowledge of the students but actually managing this gap as part of the teaching strategy. The design of PTB allows trainees with different levels of prior knowledge to select the scenario that best meets their knowledge level and ensure that everyone succeeds in the simulation in the chosen level of difficulty. Implications for lecturers suggest they should not treat SBT as a stand-alone teaching technique. SBT is found to be most effective when supplementing current educational methods such as lectures (Salas et al., 2009). Following our research findings on the impact of prior knowledge level on knowledge enhancement, lecturers should ensure that students possess the required theoretical knowledge before the SBT session.
Future research can build upon these findings by identifying additional contextual variables that may moderate the effectiveness of SBT on learning enhancement; collecting data in various cultures, academic levels, and with a larger sample size; and using different simulators in other areas of management education. Studies can also focus on setting a more accurate level of prior knowledge required for an effective SBT session.
