Awardees: Marshall Thomas (HMS), Selen Turkay (CADM-VPAL), and Michael Parker (HMS)

Summary: Awardees will explore the best-performing sequences of instructional materials in both controlled studies and in the context of real online courses.

Marshall Thomas (HMS), Selen Turkay (CADM-VPAL), and Michael Parker (HMS) are studying the impact of the composition and sequence of instructional materials in online courses on students’ learning and subjective experiences, such as engagement, mind-wandering, and enjoyment. Their long-term goal is to develop a clear foundation of evidence to inform the following practitioner questions: How should I sequence instructional materials? What should I make mandatory and what can be left as optional? What is the marginal benefit of adding a particular instructional resource to an existing course?

The awardees drew materials directly from the Immunology course of HMX Fundamentals ­ fully online courses directed at a variety of learners. They recruited participants from Amazon Mechanical Turk, a market for tasks requiring human intelligence. Using randomized controlled trials, they studied the effects of varying instructional activities (namely, assessment and a step-­through interactive) on participants’ learning and subjective experiences. They found that participants were more likely to drop out when they were in conditions that included assessment – an undesired effect that could potentially offset some of the known benefits of assessment. Moreover, assessment with minimal feedback (correct answers only) led to the lowest subjective ratings of any experimental condition. These negative subjective effects of assessment were mitigated by the addition of assessment explanations or a summary interactive. Moreover, the awardees found that prior knowledge and self-rated confusion were predictors of learning (as measured by follow-up test scores). Their results have a clear implication for course developers: the addition of explanations to assessment questions is a simple way to improve students’ experiences in online courses.

The awardees have shared these findings within Harvard at education-focused events such as HMS Medical Education Day and the joint retreat between Preceptors and Curriculum Fellows, and have presented the results at a national education research meeting. They are submitting a publication for peer review to share their findings more broadly. In future work, they plan to study the effect of adding other components or sequences of materials to online courses.