International Journal of Virtual Worlds and Human Computer Interaction (VWHCI)

ISSN: 2368-6103

1. Introduction

Capstone courses for undergraduate students provide undeniable learning benefits, and are considered among the best high impact learning practices in undergraduate education [1, 2, 3]. In addition to providing a "springboard for connecting learning to the world beyond campus", [1] capstones are an excellent way to assess the success of student learning in the major [4, 3, 5, 6, 7, 8], and many institutions increasingly require capstone courses or experiences for senior level, undergraduate students. According to the 2011 National Survey of Senior Capstone Experiences, 268 of 276 responding institutions offer a capstone experience [9], and the effectiveness of capstone experiences at increasing levels of engagement and to create deep learning has been documented in national surveys [1]. In fact, the 2008 AAC&U's LEAP studies on High Impact Educational Practices finds that one of the most effective ways to enhance student engagement and increase student success is to "make it possible for every student to participate in at least two high-impact activities during his or her undergraduate program " one in the first year and one later on such as a culminating experience [1].

But what are the most effective delivery methods for capstone courses, and can they, in a time of high demand for online learning, be as effective when offered in non-traditional formats as when they are taught in traditional classroom settings? What do student learning outcomes indicate about some of the most effective formats for delivering a capstone learning experience? In addition, can answers to these questions inform programmatic curriculum design?

In 2008, The University of North Dakota (UND), a comprehensive research university with over 15,000 students enrolled in undergraduate, graduate and professional and medical programs, put into place its revised General Education program with an emphasis on encouraging students to experience general education learning goals throughout their entire undergraduate experience. Several new requirements were added to the program that has now been titled Essential Studies, the most exciting of which is a senior capstone experience. This decision has been especially noteworthy since only approximately 17.9% of surveyed institutions serving over 3,000 students offer capstone courses for all undergraduates [9]. To help provide capstone experiences, a three-credit interdisciplinary capstone, Writing Across Disciplines, was developed in the Humanities & Integrated Studies unit at UND. Meant to bring students from many varied majors together to dialogue on the goals and methodologies of their disciplines and to examine where and how these goals intersect, this capstone became the first cross-disciplinary capstone offered at UND and the only capstone to be offered in three different ways: online, on-ground, and in an on-ground/online hybrid format. At the University of North Dakota, traditional face-to-face classes are referred to as "on-ground"; courses entirely offered in a non-face-to-face, asynchronous, online format are referred to as "online"; and courses that blend both face-to-face group learning with asynchronous, non-face-to face, online components are referred to as "hybrid" courses.

To determine which platform was most effective for helping students achieve their learning goals, a pilot study of the learning outcomes was created to compare the three formats and try to judge the preliminary effectiveness of each. All three courses are taught by the same professor, and use the same texts and assignment prompts and emphasize the same learning goals. However, necessarily, the methods for students interacting with each other, with the texts and topics, and with the assignments themselves differ according to each delivery method. Direct assessments have been done on student learning outcomes from each type of course delivery.

Additionally, by analyzing assignments and by comparing those results with student feedback from these courses, instructors are beginning to better understand which course formats and which types of assignments are the most effective at helping students hone capstone level abilities in writing, critical thinking, and integrative learning. Though not all capstones may strive to help students meet these same academic goals, the findings here should help instructors think about how best to create capstone assignments and to develop methodologies that are meaningful and effective for the common capstone student learning outcomes. Additionally, the findings from this study provide insight into larger, programmatic curriculum design in lower level courses and will help in the formulation of the learning outcomes for this course and other capstones. It will allow the faculty to construct an assessment program that allows them to determine most effective methods of course formats, to add to the dialogue around effective methods for teaching highly student centered classes/learning community based classes and to contribute to the growing body of knowledge about the effectiveness of online teaching.

Another unique aspect of this study, is that Humanities & Integrated Studies is a department that believes strongly in student centered learning, interactive learning communities, and high levels of faculty engagement with students.  These are hallmarks of all of the classes taught in this unit. So, if a capstone course is offered in a seemingly less-personal, less immediate format, like an online format, could it foster the same levels of student learning and could it be as successful as its traditional counterpart? Answering this question could provide units that emphasize a learning community approach with cost effective ways to reach more students while still maintaining the outcomes that student-centered, learning community practices engender.

2. Background

The effectiveness of online courses, and distance education in general, has been a topic of intense study for the past several decades.  While early distance education experiences often relied solely on audio or video recordings, textbooks, workbooks, and other "low tech" strategies, today's online learning applications provide diverse learning opportunities through applications such as course management systems, video streaming, educational games, chats, wikis, blogs, and personal response systems.  It has been suggested that online educational technologies are often better than conventional instruction approaches [10, 11].  However, the historical context of studies investigating the effectiveness of online or distance education should be taken into account when interpreting results of those studies.

Although numerous reports describe instructional approaches taken in online capstone courses [12, 13, 14, 15], relatively little is known specifically about the effectiveness of online senior-level integrative capstone courses at helping students achieve stated learning goals and little has been done to look at learning outcomes in virtual or hybrid learning communities [16, 17].  Most studies of online learning in general indicate that learning outcomes in purely online courses, or even hybrid courses (ones that incorporate components of online and traditional on-ground, face-to-face interaction) tend to be at least comparable to those in purely on-ground courses.  For example, a recent meta-analysis examined the effectiveness of purely online and hybrid courses compared to traditional on-ground (face-to-face) courses [18].   Ultimately, this analysis focused on 45 studies published between 1996 and 2008.  Online learning (including purely online as well as hybrid courses) on average yielded higher learning outcomes than solely face-to-face instruction.  Furthermore, the hybrid courses showed a larger advantage compared to pure face-to-face instruction than did pure online courses.

Other meta-analyses based on earlier distance education literature reported results similar to those mentioned above [19, 20, 11].  In addition, asynchronous instruction, with a lag between instruction periods (posting of information, assignments, etc.), had a small positive effect on student performance whereas synchronous instruction (where instruction and student response occur in real time) had a small negative [19, 21].

There has been a shift in online learning environments over the past few decades from information transmission to more interactive and collaborative learning with the online technologies that have become increasingly available.  The term "online learning communities" refers to coordinated face-to-face and virtual interactions among a cohort of learners led by one or more instructors [22, 23].  It has been suggested that effective learning, especially when dealing with complex, multidimensional issues, takes place best in learning communities and that online learning can effectively provide such communities [23, 24, 25]. This pilot study seeks, in part, to make initial determinations about whether or not new, more collaborative technologies could be utilized in online and hybrid classrooms to produce acceptable or, perhaps, even improved, student learning outcomes.

3. Course Formats and Learning Goals

The "Writing Across Disciplines" capstone was created to meet the emerging criteria of UND's Essential Studies (general education) capstone requirement and is, thus, a course that provides students in their senior year, who have completed basic required writing courses, with an upper-level, intense writing experience that focuses on the methods and strategies of writing in the Humanities, Social Sciences, and Sciences. It also asks students to engage in the process of integrating interdisciplinary materials and writing tactics as well as to formulate written responses to topics of current concern in individual and in collaborative assignments. Students enrolled in this course come from a wide range of majors including (but not limited to) Communications, Forensic Science, Physical Education and Exercise Science, Recreation and Tourism Studies, Physics, English and General Studies. They may or may not have had prior experience in student-centered interdisciplinary classes. As was stated previously, the course is taught in three ways, but all courses use the same texts and assignments and emphasize the same learning goals, which are that students will:

• Appreciate and respond to diverse audiences and diverse perspectives
• Use reading and writing to develop critical thinking skills through inquiry, learning, thinking, communicating and collaborative work
• Understand writing as an on-going process
• Use research to develop, support and enhance ideas, while using the appropriate writing conventions required by different disciplines
• Understand the value of multiple perspectives regarding complex issues and problems

The courses necessarily differ, however, in how students work to meet these learning goals and how they respond to each other, the texts, and the assignments. This table below provides an overview comparison of the course structures, and a detailed discussion of each format follows.

	Face-to-Face time	Online usage	Instructor Participation/Facilitation	Writing as primary student interaction	Oral discussion as primary student interaction	Student to student contact	Real Time interface	Student Collaboration
Online		X	X	X		X		X
On-ground	X		X		X	X	X	X
Hybrid	X	X	X	X		X	X	X

3.1. The On-ground Version

The original format for the course embraces traditional classroom practice, bringing a small group of senior students from a variety of majors together to read, discuss and write about topics and readings in each of the main disciplinary areas of Humanities, Social Science, and Science.

The on-ground version of the course integrates short readings from the varied disciplinary areas including a short novel, a vignette, non-fiction articles and a play in addition to other short readings. The assignments involve in-class discussions and activities, in addition to both informal and formal writings which take the forms of in-class short writing assignments, personal reflection, letters to the author, individual and collaborative presentations, and individually and collaboratively written formal essays as well as student-to-student peer review memos and face-to-face conferences. In short, the intent is to introduce a variety of writing formats, encourage critical reading and reflection, encourage students to consider their readings from alternative perspectives and foster a collaborative classroom environment focused on developing strong written and oral communication skills. Instruction in this format is for this course is synchronous, where instruction and student responses occur in real time.

3.2. The Online Version

The need for non-disciplinary capstone courses that could be taken online soon developed. The faculty and staff of Humanities & Integrated Studies decided to hire an expert online educator and online course developer to translate the class material into this format and to become the main instructor for both the on-ground and online versions.

While the online version is similar in content to the on-ground version of the course, the course delivery is totally asynchronous, with a lag between instruction and student response, and it is solely focused in the online course forum (Blackboard).  Students who may be located in various other states or countries never meet face-to-face and, therefore, the necessary interaction is facilitated solely in the online forum. The course integrates the same short readings as the online courses from the varied disciplinary areas including a short novel, a vignette, non-fiction articles and a play in addition to other short readings. The assignments involve both informal and formal writings which take the forms of blogs, personal reflection, letters to the author, individual and collaborative presentations, more structured weekly writings, and individually and collaboratively written formal essays as well as responses to classmates' writings and student-to-student peer review memos with a strong emphasis placed on student interaction in the online forum. In short, the intent is to introduce a variety of writing formats, encourage critical reading and reflection, encourage students to consider their readings from alternative perspectives, and focus on improving digital communication skills through the introduction of digital collaboration tools. Through the online version of the course, the students obviously must rely on written communication and so are constantly thinking about their own writing, reading their classmates' writing, and are writing continuously throughout the entire term.

3.3. The Hybrid Version

The online course was very successful, with students demanding and filling multiple sections. As faculty began to consider the success of the course, it was noted that some of the mechanisms used in the online course, like the blog postings and challenge writing assignments seemed to encourage and sharpen certain abilities such as writing itself, counter-argumentation, and reflection on other's writing styles in relation to their own. Could it be that the necessary methods of online learning actually enhance overall learning in the course? With a strong pedagogical belief in maintaining the close student-to-student and student-to-faculty interactions of a learning community, instructors decided to experiment by creating a hybrid course where students would meet face-to-face for three week blocks with additional online writing assignments alternated with work solely done online for three week blocks, thereby combining what they believed to be the best of both online learning and on-ground classroom practices.

The hybrid version of the course integrates both the face-to-face interaction in the classroom through discussion, individual and collaborative presentations, and face-to-face/memo peer review with the same short readings from the varied disciplinary areas including a short novel, a vignette, non-fiction articles and a play in addition to other short readings. The assignments involve in-class discussions and activities as well as both informal and formal online writings which take the forms of blogs, personal reflection, letters to the author, more structured weekly writings, and individually and collaboratively written formal essays as well as responses to classmates' writings and student-to-student peer review memos, creating space for both synchronous and asynchronous instruction and learning. In short, the intent is to introduce a variety of writing formats, provide opportunities for writing and reading one another's writings, encourage critical reading and reflection, encourage students to consider their readings from alternative perspectives, while still capitalizing on the benefits of face-to-face interaction.

4. Comparing Direct Student Learning Outcomes: Methodology

Though capstones are regularly used, and often solely created, as assessment mechanisms for learning in the major or for programmatic review of majors or of general education, until recently, "research on capstone experiences is scant" [1]. Just as in any course, however, direct assessment of student learning artifacts is essential to identifying best practices in the classroom and to studying the overall effectiveness of the course to meet the prescribed learning goals, and, of course, to revise practices or materials as needed to help students better meet these goals [8, 26].  In the case of these three formats, if the mechanisms for delivery or response in one version indicates better learning outcomes than another, it would be important to revise the other versions of the course, where possible, to take advantage of these mechanisms. This pilot study was approved by the University of North Dakota's Institutional Review Board.

The online version of the course has been offered multiple times. The on-ground and hybrid version have each been offered once. This pilot study utilized direct measures for learning outcomes using the pre-/post-design that has become the standard for college effective studies [27]. For each version of the course, including two sections of the online course, formal writing assignments from the beginning and from the end of the semester were collected, and a random sample of one quarter of the papers for each assignment, each semester, each version were selected. The AAC&U rubrics for Writing, Critical Thinking, and Integrative Learning were selected for assessment use since the course grading rubrics and assignment formats emphasized learning outcomes based on these rubrics [28].  Each rubric ranks multiple criteria in each category on a scale from 0-4. A student-friendly version of these rubrics was provided to students at the beginning of the semester, used for each evaluation, and utilized by the students in each peer-editing situation, so they were thoroughly familiar with the evaluation criteria.

For the purposes of this pilot study, an outside expert who utilizes similar teaching techniques, uses the same assessment rubrics, and was instrumental in overall course development, was selected to assess the papers.  All identifying information was removed from the papers and several were selected for a norming session, where the course instructor and the outside expert applied the rubrics and compared and normed results to ensure that the rubrics were being accurately applied.

Formal papers were selected from the beginning and from the end of each semester. The first assignment was taken from the beginning of the semester and asked the students to begin with an important quotation from a selected Humanities text and explore the meaning and relevance of that quote in terms of the work, of their own experience, and of the wider world. The second assignment was the final assignment for the semester and required that students explore an issue of particular concern or interest encountered in the course and examine that topic from an interdisciplinary perspective, demonstrating how experts in different fields would address the topic and drawing conclusions with regard to those disciplines.

Papers were numbered, recorded by section, and then randomly shuffled by a third party before being distributed to the expert reviewer to eliminate knowledge of course version. The papers were all assessed and ranked according to each of the criteria (writing, critical thinking, and integrative learning). Results were averaged overall in each category and comparisons were made among course versions as well as between initial and final assignments within each course.  One-way analysis of variance (ANOVA) and Mann-Whitney analyses were used to test for differences among average rank responses. 

5. Results & Discussion

Average rankings of assessment criteria reveal noteworthy differences in student performance (and gains) among the online, on-ground, and hybrid courses (Figure 1).  Averages for the first paper did indicate lower than expected levels in all three areas (averages on this assignment ranged from 2.04±0.40 to 2.26±0.30 (mean ± standard error) on a scale of 0 to 4).  Interestingly, performance on the first paper in all three areas was statistically comparable among all course versions (ANOVA and Mann-Whitney tests, p > 0.05).  These results suggest that regardless of type of course enrolled in, students do not come in to the capstone experience with adequate experience and skills, a finding that corroborates the results of other capstone assessments [29, 30, 3, 4].

Comparisons of first and last papers reveal a demonstrated increase in learning outcomes in all course versions.  Average rankings increased in all three areas of writing, critical thinking, and integrative learning with students nearing the rank of 4 or "capstone" benchmark on the final paper for learning outcomes in each area as defined by AAC&U.   Interestingly, average rankings for the final paper were significantly higher for all three skills than on the first paper in the fall 2011 (online) course (Mann-Whitney test, p < 0.05).  This trend was also observed in the hybrid course as the average rank for thinking skills was significantly higher for the last paper than the first paper (Mann-Whitney test, p < 0.05).  Although the fall 2012 (online) and the on-ground courses also showed increased rankings by the end of the semester, they were not statistically significant (Mann-Whitney test, p > 0.05).

Performance comparisons on the final paper reveal noteworthy differences among the three course versions.  Average rankings of both online courses (Fall 2011 and Fall 2012) for writing, thinking, and integration skills were found to be statistically comparable (Mann-Whitney test, p > 0.05).  Therefore, performance results were consistent between the two semesters the online version was offered.  Overall, average rankings for the online courses tended to be higher than both the on-ground and hybrid versions while rankings for the on ground course tended to be the lowest of the three course versions (Figure 1).  Average rankings for the hybrid course were more comparable to the online course than the on-ground course.  Although differences were observed among average rankings on the final paper, only two comparisons showed significant differences (namely writing and integration skills were significantly higher in the fall 2011 online course than the on-ground course (Mann-Whitney test, p < 0.05).

Though, the factors of student background, GPA, and gender were not controlled for, all courses had the same number of students, and there were a predominance of similar majors across all versions. Additionally, students were selected from cohorts that were contemporary, meaning that all students underwent similar university academic experiences within a contiguous timeframe.  In this pilot study, only materials from one of each of the course versions was used, since the hybrid and onground versions had only been taught once. Thus, comparing one of each of the courses taught within the same time span would, it was believed, yield the best data for guiding future assessment work.

That the online version of the course was at least equal to, if not better than, the on-ground version in producing improved student learning outcomes in all three learning areas, reflects national findings [2, 31, 32].  The nature of the online learning environment has been one that has come under extensive scrutiny in an attempt to compare the online learning experience with the traditional on-ground experience [2, 31, 32].  While some studies have found "no significant difference" in the quantitative learning outcomes of online and on-ground students, the quantitative data shows something different with online students indicating greater satisfaction and success with "well designed" online course experiences [2, 3, 31, 32, 33, 34, 35]. In this pilot study, these quantitative findings were reflected across the board with student self evaluations given at the end of the semester.  Student pre- and post-writing assignments and reflective comments illustrate recognition by the students of the progress made in all areas of writing, critical thinking and integration.

With the proliferation of technology, the Internet and online learning tools, it is possible to create a richer, more interactive and engaging environment for students to develop writing skills and complete capstone experiences. However, this plethora of resources and new technological advancements require substantial instructional design to integrate structure, course material, increase familiarity between the students and with the tools available [3, 6, 32, 35, 36]. This can be effectively translated to online writing courses as well [36] including capstone courses. 

The potential for increased benefits in the online version may also be due to the asynchronous nature of the course. When students can approach course material and engage in course work at the most optimal times for their learning styles and lifestyles, they may be more likely to produce work that is better than it would be in synchronous learning environments when work must be accomplished on the instructor's or institution's timeframe and not the students. Additionally, the asynchronous format is "much more conducive to the expansion of learning time, thereby providing more opportunity for student reflection and response" [6, 34, 35, 37, 38].

The focus on the written word in the online learning forum emphasizes reading and writing, as opposed to speaking and listening in the on-ground and hybrid formats, which impacts the way that learning occurs.  Students in an online learning forum cannot necessarily rely on others to carry the discussion because each student is required to complete the written assignment and post responses to their classmates [36].  The "piggy back" nature of the discussion allows for development and recording of asynchronous discussion in the online environment over time for the students to revisit and reconnect with at any time during the term [38]. Additionally, this kind of layering or building up of communications over time to develop each assignment requires that students read each others' writings and, therefore, necessarily become more deeply familiar with each others' ideas in order to formulate individual responses and write assignments.  This deep learning and necessary familiarization with others' ideas accomplishes, to a strong degree, what in-class socialization does: it allows students to form a learning community. They become connected through the process of reading and responding to each others' postings. Of course to have this occur, the course must be well constructed so that it requires this constant reading and revisiting of others' work [34, 35, 38].

6. Plans for Expansion of the Study

The results of this study are indicative of the need to pursue a larger-scale program of assessment. Though preliminary findings indicate that online and hybrid writing-intensive capstones provide students with high quality learning experiences, continuing the study and drawing from multiple sections over time will help faculty determine if these findings are robust if they persist.

In addition to studying the student learning outcomes via assessment of learning artifacts, additional data about students would help lend perspective to the findings. In the next, larger scale assessment, information on student demographics will be added, including information on GPAs, gender, and previous experience in writing and discussion-based courses. Analyses of these additional variables may help pinpoint some of the factors associated with effective demonstration of learning outcomes at the capstone level.

Additionally, since, in this pilot study, the onground and the hybrid courses had only been offered one time, materials drawn from subsequently taught courses should yield more robust information. Once the hybrid course is more systematically taught, the possibilities of any unevenness due to instructor issues (e.g. being new to the hybrid format, etc.) will be more easily controlled for. The faculty now plans to collect data from all sections of all formats of the course, choose random samples, and evaluate them so that a more robust pool of information can be compared.

Additionally, to further pursue the effect of online formats on learning community engagement, a survey of student perceptions of learning and engagement will be conducted and analyzed, using responses from students enrolled in all three course formats. This mixed methods approach will provide valuable insight for other institutions that wish to create capstone experiences in an online learning environment.

7. Conclusion

An overall analysis in this pilot study indicates that the learning outcomes in the hybrid course tended to equal those in the online course in most areas, and results would indicate that offering the on campus version of the course in the hybrid, rather than the full on-ground version, would be more beneficial to students. The hybrid version of the class allows for the integration of the different in-class and online activities and strategies to reinforce and supplement the objectives and goals of the capstone experience.  For many students this mix encourages consideration of the course material beyond the class period and keeps students more engaged over the entire course of the term as they prepare the online assignments than the solely on-ground version of the course, which does not include this component. Additionally, the emphasis placed on writing, online discussion, and peer review through the online portions of the class provided opportunity for improved and continuous writing in a variety of informal and formal formats over the course of the term, as well as opportunities for revisiting and reflecting on past writings. Students also maintain a visual record of their writing progress over the entire term as they develop writing, critical thinking and integration skills.  By revisiting past writings and evaluating their own progress, students are able to see improvements in their own writing, while also see improvements and stylistic differences in their classmates' writings. 

In all three versions of the course, students' averages for writing, critical thinking, and integrative learning were lower than expected for the first assignment. This indicates that more reverse programmatic curriculum planning needs to occur at every level of Essential Studies course work to provide students with more opportunity to develop these skills before reaching the capstone.

As for the best format for offering a capstone course that emphasizes writing, critical thinking and/or integrative learning skills for student learning goals, the online and hybrid method offer at least as good a learning experience, if not perhaps, an increased benefit, when compared to the on-ground course. Though, as indicated, a further, more detailed assessment will be undertaken, these results indicate that in some circumstances students can gain as much learning benefit, if not perhaps more, from well-constructed online or hybrid learning experiences as they can in a face-to-face class.  The creation of true learning communities in the virtual world does seem to be effective and may even provide students with more opportunities and space for reflection, leading them to greater learning gains overall.

References