Colleges and universities are under pressure from a variety of stakeholders to demonstrate evidence-based, authentic assessment results. The purpose of this case study was to describe one private university's Master of Science in Project Management (MSPM) degree program's use of the electronic portfolio (ePortfolio) capstone project as a programmatic assessment instrument. The ePortfolio provided a multidimensional assessment tool, showcasing mastery of the National Project Management Standards and Practices through evidence (artifacts) collected during the coursework. Trends identified during the annual review revealed a disparity of program outcome achievements for students in the online program compared with students in the traditional classroom format. These trends were later labeled dependencies. As evidenced by the ePortfolios, the student's success in the program was too dependent on the delivery mode, campus location, and the professor teaching the class. The dependencies were analyzed and intervention strategies implemented to reduce the effect of the dependency. A master ground-template (electronic course shell) was created for each of the six project management core courses and the capstone course for use in the traditional classroom format. The ground-template serves as a mechanism to provide consistent information to the professors and students, regardless of location. Through the collaborative inputs from content and design experts, the focus of the ground-template is providing a content-rich resource for the professor. This approach provides a platform for consistency of a curriculum, regardless of delivery mode, campus location, or professor.
The literature review provides context for the decision to use ePortfolio as a means of programmatic review for a Master of Science in Project Management program at a private university. The review starts with a brief background of the transition from paper-based portfolios to ePortfolios. The review describes how ePortfolios are being used within undergraduate and graduate education and shows that ePortfolios have gained traction, particularly within programs to show personal development for teacher education and medical training. ePortfolio literature from the higher education field is summarized and the use of ePortfolios as a tool for program evaluation and assessment is described. The review ends with a brief description of the literature criticizing the use of ePortfolios for student-centered learning and recommendations from the literature for implementing a successful ePortfolio program.
From Paper to Electrons
The first use of student-developed portfolios dates back to the 1960s (Ehley, 2006). Before that, portfolios were used by artists and designers as a means of collecting and displaying their work (Chatham-Carpenter, Seawel, & Raschig, 2009). Portfolios were very popular, beginning in the 1960s through the early 1990s, because of a widespread belief within the liberal education movement that portfolios provided a means of authentic assessment. According to F. Leon Paulson and Pearl R. Paulson, pioneers in the use of portfolios for educators:
A portfolio is a purposeful collection of student work that exhibits the student's efforts, progress, and achievements in one or more areas. The collection must include student participation in selecting contents, the criteria for election, the criteria for judging merit, and evidence of student self-reflection. (Paulson, Paulson, & Meyer, 1991, p. 60)
Ehley (2006) provided evidence of extensive use of paper-based portfolios for authentic, reflective, and normative assessments between the 1960s and 1990s. However, Yancey (2009) asserted that paper-based portfolios were used in a limited manner for summative assessment in a single course. Paulson et al. (1991) argued that a portfolio, by definition, featured a student as “a participant in, rather than the object of, assessment” (p. 63).
The popularity of paper-based portfolios waned in the early 1990s due to national education reform toward standards-based assessment (Ehley, 2006). Portfolio use surged in the mid-1990s as programs shifted toward storing documents in electronic format. ePortfolios became very popular among higher education institutions as a “tool to enhance learning, conduct assessment, meet standards, and increase student employability” (Chatham-Carpenter et al., 2009, p. 438). The swing from favor to disfavor back to favor is consistent with Gartner's hype cycle (O'Leary, 2008). Looked at through hype cycle lens, the portfolio (as a technology or tool) may have been in the trough of disillusionment. The transition to electronic portfolios could be viewed either as the move to the slope of enlightenment or as a trigger for the ePortfolio hype cycle (see Figure 1).
The 2006–2009 Gartner Hype Cycles for Higher Education showed ePortfolios sliding into the Trough of Disillusionment (Zatrosky, Harris, & Lowendahl, 2006). The 2010–2011 Gartner Hype Cycles depicted ePortfolios on the Slope of Enlightenment, indicating that ePortfolios are being perceived in new and useful ways (Lowendahl, 2011).
The Role of ePortfolios
With the advent of electronic portfolios (ePortfolios), student portfolios adopted a greater role in documenting learning within and across courses and experiences, sometimes beyond the bounds of the formal educational environment (Yancey, 2009). Butler and her colleagues (2006) documented several purposes for the use of portfolios: learning, professional development, normative assessment, summative assessment, and career advancement. Chatham-Carpenter et al. (2009) described four purposes for ePortfolios: facilitating reflective learning, showcasing career skills, showcasing professional standards, and assisting with program review and assessment. ePortfolios are used in so many ways that Barrett (2007) argued that “the term portfolio should always have a modifier or adjective that describes its purpose” (p. 436).
The National Learning Infrastructure Initiative defined an ePortfolio as: A collection of authentic and diverse evidence, drawn from a larger archive representing what a person or organization has learned over time, on which the person or organization has reflected, and that is designed for presentation to one or more audiences for a particular rhetorical purpose. (The National Learning Infrastructure Initiative as cited in Barrett, 2007, p. 438)
Educational portfolios expanded upon traditional portfolios through learner collection, reflection, and selection of educational artifacts. Unlike traditional portfolios that highlight the best artifacts, educational artifacts were chosen to demonstrate growth and change over time (Barrett, 2007). The focus on student reflection and selection of artifacts was informed by the literature on reflective practice (Schon, 1987), preparing students to adapt prior practice to new situations and to articulate experience and growth. This process extends beyond reflective commentary, encouraging students to exercise higher order thinking to “learn what one has learned” (Wang, 2010). Through development of an ePortfolio structured to demonstrate PMI competencies, learners transition beyond their experiences as students and project themselves as professionals. The model of reflective practice also initiates a process for future professional development, where critical inquiry draws on insight into personal learning and one's assumptions about professional practice (Webster-Wright, 2009).
The Benefits of ePortfolio Use
The benefits of ePortfolio use include development of narrative skills to identify strengths, reflection on personal development, and formulation of professional identities (Graves & Epstein, 2011). A majority of students view ePortfolios as a useful learning tool and consider it to be an important assessment component in a course (Yusuf & Tuisawau, 2011). Portfolios help students focus thinking, translate theory to practice, document progress over time, improve communication and organizational skills, recognize a prior knowledge, and identify learning outcomes (Butler et al., 2006). According to Buckley, Coleman, and Khan (2010), the highest quality studies (quality was based on analysis against Kirkpatrick's hierarchy) showed that properly implemented ePortfolios improved the integration of theory with practice, encouraged self-awareness and reflection, and supported “students facing difficult emotional situations” (p. 187). Barrett (2007) described the value-added benefits of ePortfolio over paper-based portfolios as archiving, linking and thinking, storytelling, collaborating, and publishing.
Online storage provided excellent accessibility for ePortfolio owners, instructors, colleagues, and employers (McCowan, Harper, & Hauville, 2005). ePortfolio development helped develop the technical skills needed to maintain online professional identities and enhanced professional networking skills (Kryder, 2011). ePortfolios allowed for the use of multimedia artifacts (Boggan & Harper, 2009). In addition to providing a digital display of professional competencies, ePortfolios enabled program administrators to conduct comprehensive assessments of curriculum and learning outcomes (Wang, 2009). ePortfolios may also be used as a form of data collection for accreditation agencies (Boggan & Harper, 2009). According to multiple authors, the field of educating teachers is the most advanced in terms of the use of ePortfolios (Buckley et al., 2009; Buckley et al., 2010; Butler et al., 2006; Chatham-Carpenter et al., 2009).
Boggan and Harper (2009) noted that ePortfolios have been used to prepare for accreditation reviews. ePortfolios were identified as an ideal tool to provide a programmatic review for a course or a program (Strivens et al., 2009) and specifically as a programmatic review tool within graduate-level programs (Moore, Tatum, & Sebetan, 2011). As described in the Chronicle of Higher Education:
The tasks of setting institution-wide goals and overseeing faculty practices and curricula “are now more in potential for alignment than they probably have ever been,” she said. “And part of that is because we now have the evidence that can be collected and shared in e-portfolios.” (Basken, 2008, p. A30)
Walvoord (2010) described three steps of assessment that coincide with accreditation reviews: (1) goals—determining what students should be able to do at the completion of a program; (2) information—developing measures of how well students are achieving the goals; and, (3) action—using the information to improve the student learning.
Criticism of ePortfolios
Although in theory ePortfolios were believed to yield benefits for learning, in practice, ePortfolio use often leads to confusion and frustration (Chau & Cheng, 2010). Deneen and Shroff (2010) explored whether the benefits of ePortfolio use outweighed the costs involved in information and communications technology (ICT) literacy. Their conclusions were affirmative, but included significant cautions regarding technology challenges that may be addressed through time, structure, and diligent effort.
An and Wilder (2010) cautioned that successful implementation of ePortfolio required significant attention to process and work flows. Dietrich and Olson (2010) reported that administrators expressed concerns over the potentially excessive amount of time required to use ePortfolios as assessment tools.
Criteria for Success
The literature has numerous research and case studies offering advice for successful implementations of ePortfolio. The recommendations are provided in diverse areas such as stakeholder management, faculty and student engagement, program organization, mentoring and training, and information and communication technology (ICT) literacy.
According to Barrett (2007), the teacher's role is critical to success. High-performing teachers (as judged by student engagement) effectively used reflection, metacognition, and other learning strategies to provide excellent feedback. Comprehensive ICT strategies, ICT skills, support systems, and collaborators were also indicative of high levels of student engagement. Deneen and Shroff (2010) pointed to the need for ICT literacy and a well-designed program structure to contribute to successful ePortfolio implementation.
Shouhong (2009) proposed a model for improving reflective learning. The model was developed using inquiry as the primary means of directing the student through the portfolio process, thereby enabling high-quality reflective learning. Tindall-Ford, Waters, and Johnson (2010) described several requirements: thorough preparation of staff and students; ePortfolio assignments planned from the ground-up, not implemented as an overlay on existing assignments; the need for key stakeholders to agree on the criteria and function of the ePortfolio system; the use of recommended training and ongoing support; a positive and supportive culture; and availability of the ePortfolio to the students after graduating.
Dietrich and Olson (2010) stressed the importance of early agreement on the purpose of assessment and the need to promote cooperation among administrators, faculty, consultants, and program reviewers. According to Hallam and Creagh (2010), open dialogue and collaboration between stakeholders are critical. Granberg (2010) advised that it is necessary to overcome the tension between using ePortfolios for summative assessment versus for ongoing reflection and learning. Buckley et al. (2009) suggested that an ePortfolio program have the following features: reasonable time demands; support to develop reflective skills; portfolios designed to reflect training requirements; specific aims and objectives; alignment with course outcomes; clear guidelines on requirements, word count, and time commitments; and that the program be delivered with plenty of time to enhance reflective skills
Colbert, Ownby, and Butler (2008) suggested that learners and faculty should be trained explicitly on process, purpose, structure, and content; mentoring should be considered, if necessary; and, evaluation rubrics should be used if the program intended to use summative assessment. An and Wilder (2010) described a preference for implementing ePortfolios using a bottom-up rather than a top-down approach. They also advised of the need for plenty of administrative support for faculty and students, systematic technical support, and shared faculty vision. Ehley (2006) recommended that the program be designed to ensure regular engagement with the ePortfolio and adequate ICT and process training for both students and faculty.
This paper describes the evolution of one private university's Master of Science in Project Management (MSPM) degree. As of fall 2011, the university had 16,619 students enrolled in undergraduate- and graduate-level programs. There are 130 satellite campuses in the United States, Europe, Middle East, and Asia. The classes are administered in five modalities: traditional classroom format (referred to here as ground courses), pure online format, video conferencing between campus classrooms, video conferencing between home computers, and a blended format merging online and classroom instruction.
The MSPM program accepted its first students in the fall of 2007. The program is administered in two modalities: online or on the ground. The ground course format is offered at ten of the university's 130 satellite campuses. Since its inception, the MSPM has graduated 124 students and another 38 students have completed the capstone and are awaiting graduation. Its current declared student body is 384.
The MSPM's curriculum recognizes Project Management Institute (PMI) as the certification body and the global authority for the project management industry. There are six sequential project management core courses, five general management courses, and a capstone course (see Appendix 1 for MSPM program objectives and course learning outcomes). The curriculum and each of the core courses were developed by certified Project Management Professionals (PMPs)®. All of the professors who teach the classes have doctorates and are PMP® credential holders.
MSPM courses were designed through a partnership between course developers (academically qualified with earned doctorates and professionally certified PMP® credential holders) and instructional designers. Based on backward design and A Guide to the Project Management Body of Knowledge (PMBOK® Guide), Table 3-1, Project Management Process Groups and Knowledge Areas Mapping, learning outcomes were developed to drive the assessments and activities. Online courses are fully asynchronous and are generated from templates that are highly structured with mandatory activities and assessments. Traditional ground courses meet synchronously during which the professor is provided with a course outline detailing the specific learning outcomes.
The MSPM culminates in a unique capstone: an ePortfolio. During the capstone course, students build an ePortfolio to showcase their mastery of the national project management standards and practices using evidence collected during the coursework. The students use A Guide to the Project Management Body of Knowledge (PMBOK® Guide), Table 3-1, Project Management Process Groups and Knowledge Areas Mapping (now referred to as the PMBOK® Guide Matrix), to shape the architecture of the ePortfolio (Figure 2).
By using the PMBOK® Guide Matrix, the students visualize subject matter organization and the industry's life cycle. This matrix creates a map tying the course work learning outcomes to real work industry applications. Throughout the six MSPM core courses, students are provided with assignments, which lead them to using established methods, processes, and practices of the profession. The transition from instructor-led activities and tests to student-centered projects represents a ‘real world' application of the project management tools and techniques. As the students progress through the coursework, they are encouraged to organize and classify the coursework based on the PMBOK® Guide Matrix. This reflective approach encourages the student to not only focus on the current course topic, but to consider how the current topic relates to the previous topics and to anticipate how it may integrate with the subsequent material. In this way, the student takes an active role in the learning process.
The MSPM has two full-time faculty members dedicated to program administration (a MSPM Program Chair and an Assistant Program Chair) and teaching. Approximately 80% of the classes are taught by adjunct professors. The faculty certification process requires an earned doctorate and a current PMP® credential.
Methodology: Program Review
During calendar year 2011, the MSPM Program administration set out to assess Program Objective 1 (PO1): ‘Use accepted practices to plan projects.' This approach was used to review a sample of twenty ePortfolios, seeking a representation of online students' work as well as a representation of the different campuses hosting the ground course format. The program administration planned to assess PO1 by verifying that each student completed at least one project plan. The project plan would be reviewed for content consistent with the PMBOK® Guide Matrix, Project Planning Process. What was discovered led to the examination of over sixty completed ePortfolios as trends emerged.
Online students versus traditional classroom students.
While reviewing completed project plans, it became obvious that the students who graduated from the online curriculum had far better artifacts in their ePortfolios than students who completed the instruction in the ground course format. The online students consistently included more robust project plans. Examples of robust project plans included comprehensive scopes of work, complex schedules, detailed budgets, and thorough human resource plans. Although the ground students produced project plans with a similar table of contents, the difference in the level of detail was evident. This observation led the program administration to broaden the scope of assessment in content and sample size. Not only was the artifact quality better in the online students' work, but some of the online students used multiple artifacts as evidence to support the PMBOK® Guide Matrix. Within the sample of online students' work, there were varying degrees of mastery demonstrated. In general, the online students produced better results when compared with the ground students. Given the same textbooks and course learning objectives, why would the online students produce better quality and quantity of artifacts when compared with the ground course students?
Campus location versus campus location.
The program evaluation's scope increased and that approach continued when evaluating ePortfolios from the ground students. The evaluators reviewed 40 ground students' ePortfolios to get a sense of the accomplishments. The results were categorized as follows: exceptional, acceptable, or weak. The categorization was based on expert judgment, considering professional presentation of the material, the ePortfolio creator's words to guide the reader, and the quality and quantity of artifacts presented. A noticeable trend emerged and pointed to specific campus locations. Most of the ePortfolios that were categorized as exceptional came from two of the ten campus locations. The ePortfolios categorized as acceptable or weak represented the other eight campuses equally. This discovery led to questions about academic advising of the students. Were some campuses advising students better than other campuses?
Professor versus professor.
Another trend was the evaluator's ability to identify specific adjunct professor instruction based on student artifacts. It appeared as if overarching program goals were integrated with varying degrees of success within each adjunct professor's classroom. Consequently, assignments directly reflected a professor's teaching style and content. This observation led to the question of whether student results were too dependent on the professor teaching the course.
Conclusions of the program review.
As a result of the comprehensive programmatic review, the following conclusions were made:
- Based on the differences between online and ground students' artifacts, consistent structure appears to be the most important predictor of higher-quality artifacts. Although there were many variables, the most prevalent was structure. The online students produced better ePortfolios because they were given assignments that required them to prepare quality artifacts.
- Ground students were more successful when exposed to higher quality academic advising on campus.
- The students' experiences were too dependent on the professor teaching each class.
These three conclusions were later labeled as dependencies. As evidenced by the ePortfolios, the student's success in the program was too dependent on the delivery mode, campus location, and the professor teaching the class. The next step in the process was to develop a series of ‘interventions' to significantly reduce the dependencies, providing a more consistent curriculum regardless of delivery mode, campus location, or professor.
The philosophy was that all students should have the same opportunity for success through the program regardless of delivery mode, campus location, or professor. What do the online students get that the ground students do not? The simplest answer is: structure. The online classes are delivered asynchronously and a course management system provides weekly content delivered in modules. Different professors teach the different sections of the same course and the students have a consistent structure. The question becomes: How do we duplicate the success in the ground courses? To address the question about academic advising: How do we provide the same academic advising detail regardless of campus? How do we capitalize on an adjunct's experience in the classroom and eliminate poorly designed assignments? After hours of debate, the ‘ground-template' concept emerged. This is a template of content delivered to the ground courses' professors and students and is a repository for information.
Program improvements as a result of reviewing the ePortfolios
In a simple description, a ground-template is an electronic course shell. Using the university's course management system, a master ground-template (electronic course shell) was created for each of the six project management core courses and the capstone course. They are a mechanism to provide consistent information to the professors and students, regardless of location. Once complete, the master ground-template would be provided to each instructor teaching that particular MSPM course prior to the start of the term.
A team consisting of the MSPM program administration, course monitors, and instructional designers met for a week to develop the master ground-templates. With the collaborative inputs from these content and design experts, the focus of the ground-template development became developing a content-rich resource for the professor. This is a considerable departure from the online course structure, which is designed and populated with resources for the student. Although there is some information available to the student, the goal is to provide instructors with high-quality resources and suggestions of proven academic value that respect the academic freedom of the individual professor. The purpose of the ground-templates is to provide curriculum consistency regardless of campus location or assigned professor.
Intervention strategies to eliminate dependencies
Intervention Strategy #1 — Mandatory Assignments
The assessment team concluded that the online structure led to successful development of artifacts for the ePortfolio. Mandatory assignments were created and embedded into each of the six project management courses' ground-templates to translate that success into the traditional classroom. If the lack of structured assignments created a dependency, providing structured assignments with proven academic value presented the students with opportunities to create quality artifacts. This intervention strategy provided a building block for consistency of curriculum regardless of delivery mode, campus location, or professor.
Intervention Strategy #2 — Instructor Resources
In addition to providing mandatory assignments, the ground-templates provide a suggested course administration format to include lesson plans, other suggested assignments, case studies, activities, and instructor resources. The program administrators recognized that some instructors might be concerned with a loss of academic freedom because of this detailed structure. The loss of academic freedom is mitigated, because the structure provided allows the professor to use his or her experience to build on the content instead of new course development. This intervention strategy provided a building block for consistency of curriculum regardless of delivery mode, campus location, or professor.
Intervention Strategy #3 — Videos
When reviewing the conclusions at a high level, program administrators identified communication as a chronic challenge. To ensure professors and students would receive a consistent message throughout the program, a series of informational videos were created and embedded into the ground-templates. This intervention strategy provides another building block for consistency of curriculum regardless of delivery mode, campus location, or professor.
A ground-template overview video (designed for the professor) communicates the purpose and intent of the ground-templates. In this video, the programmatic review process was explained and the findings identified. This provides the instructor with the logic behind creating the ground-template and sets expectations for course administration, lessening the dependency on the professor.
The program orientation video is not a new addition. It was created when the MSPM was initially launched and was required for new students, but the message was not being received. This program orientation video that was given added emphasis by placing it into the ground-template to be viewed every term. The dependency on academic advisors will be reduced by reminding the students about the structure and flow of the program every term.
The ePortfolio overview video provides the students with a preview of the capstone deliverable. This video serves multiple purposes. The intended audience is the student, but because the instructor will also view it, the instructor will get a better understanding of the capstone. The video is a demonstration of an ePortfolio, which focuses on what it looks like, its purpose and intent, and sets the stage to tie course outcomes to the capstone early in the student's academic journey. This video lessens the dependency on delivery mode, campus location, or professor.
Each course has its specific orientation video. The course monitor created the video, which provides an overview of the course content and also sets expectations for course deliverables. This video serves multiple purposes. The intended audience is the student; but, because the instructor will also view it, the instructor will get a better understanding of the course goals and a reminder of how that course fits into the capstone. The video ends with a reminder to the student to retain all of the artifacts for the capstone. This video lessens the dependency on delivery mode, campus location, or professor.
Intervention Strategy #4 — ePortfolio Resources
Although this strategy cannot be tied directly to one of the three original conclusions, based on the improvements made to the courses, an ePortfolio Resources section was created. In this section, students are provided with two exemplar ePortfolios that were created by previous graduates of the program. These examples provide not only an illustration of what a completed ePortfolio looks like, but the new students will see examples of exceptional artifacts created by actual students while they participated in the program. In addition, detailed instructions on how to create the ePortfolio structure are available in this resource section.
Intervention strategies improve online courses
Although the programmatic review results generated intervention strategies for use in the ground-templates, some of the intervention strategies added value for the online classes. The course orientation videos and the ePortfolio overview video were added into the online courses to add emphasis about the program capstone. Additionally, the ePortfolio resources section was embedded into every online course. The online students can see a completed ePortfolio and review examples of exceptional artifacts created by actual students while they participated in the program.
The ePortfolio is an extremely powerful programmatic assessment instrument because it contains student-created artifacts for examination. In this case study, the MSPM graduates' ePortfolios from the online program were compared with the graduates of the results of the MSPM ground program. Trends emerged that were later labeled dependencies. As evidenced by the ePortfolios, the student's success in the program was too dependent on the delivery mode, campus location, and the professor teaching the class. The dependencies were analyzed and intervention strategies were implemented to reduce the dependency. The results of the program review culminated with implementing intervention strategies to create a venue where all students will have the same opportunity for success through the program regardless of delivery mode, campus location, or professor.