The rapid growth of online education (Allen & Seaman, 2013) brings questions about how best to maintain quality in teaching and ensure effective learning. One way to assess quality is through reliability and validity measures of student scores (Norman & Eva, 2010). We assessed the reliability and validity of a component of a new, online Masters of Sciences in Health Sciences Education Program for recent graduates and health professionals. The first required online course for students, entitled ‘Learning & Curriculum,’ exposes students to the basic principles of human memory and learning and challenges them to incorporate these principles into their own learning. Specifically, students are assessed in part on their participation in discussion posts to carefully designed ‘points to ponder’. An assessment of student scores showed an acceptable level of variance, with the greatest source from students (0.73). Inter-rater agreement was also high at 0.73 across all topics of discussion. Students also benefitted from the involvement and feedback of instructors as student scores showed an increasing trend over time. These scores also correlated positively with measures of performance from other components of the course (0.3). The purpose of our presentation will be to first share information about the planning of this nascent program and our application of successful teaching and learning strategies (Dunlosky et al, 2013) to an online format. Importantly, we will discuss our analyses and the interpretation of those results. We hope to engage participants in discussions of measuring impact in education and how best to interpret data.
Allen, I. E., & Seaman, J. (2013). Changing Course: Ten Years of Tracking Online Education in the United States. Sloan Consortium. PO Box 1238, Newburyport, MA 01950.
Norman, G. R., & Eva, K. W. (2010). Diagnostic error and clinical reasoning. Medical education, 44(1), 94-100.
Mayer, R. E. (2003). The promise of multimedia learning: using the same instructional design methods across different media. Learning and instruction, 13(2), 125-139.
Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4-58.
