pdf
Published
2021-02-26
Issue
Section
Education
License
Copyright (c) 2021 Michael Peeters, M Kenneth Cor, Sai Boddu, Jerry Nesamony
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright of content published in INNOVATIONS in pharmacy belongs to the author(s).
Validation Evidence from using Generalizability Theory in a Basic-Science Course
Reliability of Course-Grades From Multiple Examinations
Michael Peeters
University of Toledo
M Kenneth Cor
University of Alberta Faculty of Pharmacy & Pharmaceutical Sciences
Sai Boddu
Ajman University College of Pharmacy and Health Sciences
Jerry Nesamony
University of Toledo College of Pharmacy & Pharmaceutical Sciences
DOI: https://doi.org/10.24926/iip.v12i1.2925
Abstract
Description of the Problem: Reliability is critical validation evidence on which to base high-stakes decision-making. Many times, one exam in a didactic course may not be acceptably reliable on its own. But how much might multiple exams add when combined together?
The Innovation: To improve validation evidence towards high-stakes decision-making, Generalizability Theory (G-Theory) can combine reliabilities from multiple exams into one composite-reliability (G_String IV software). Further, G-Theory decision-studies can illustrate changes in course-grade reliability, depending on the number of exams and exam-items.
Critical Analysis: 101 first-year PharmD students took two midterm-exams and one final-exam in a pharmaceutics course. Individually, Exam1 had 50MCQ (KR-20=0.69), Exam2 had 43MCQ (KR-20=0.65), and Exam3 had 67MCQ (KR-20=0.67). After combining exam occasions using G-Theory, the composite-reliability was 0.71 for overall course-grades—better than any exam alone. Remarkably, increased numbers of exam occasions showed fewer items per exam were needed, and fewer items over all exams, to obtain an acceptable composite-reliability. Acceptable reliability could be achieved with different combinations of number of MCQs on each exam and number of exam occasions.
Implications: G-Theory provided reliability critical validation evidence towards high-stakes decision-making. Final course-grades appeared quite reliable after combining multiple course exams—though this reliability could and should be improved. Notably, more exam occasions allowed fewer items per exam and fewer items over all the exams. Thus, one added benefit of more exam occasions for educators is developing fewer items per exam and fewer items over all exams.
