Automated scoring of student videos in medical education: a comparison between a large language model and expert evaluation
Journal of Microbiology & Biology Education · 27.08.2026 · 27 (2)
DOI: 10.1128/jmbe.00010-26 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Microbiology & Biology Education, erschienen am 27.08.2026. Verfasst von Yavuz Selim Kıyak und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medical Education and Informatics, Faculty of Medicine, Gazi University, Ankara, Turkiye. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 14 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Video-based assignments are used in medical education, yet expert scoring is time-intensive. Large language models (LLMs) offer scalable alternatives, but their validity for evaluating multimodal student work is uncertain. We examined whether a state-of-the-art LLM (Gemini 2.5 Pro) could approximate expert scoring of medical student video presentations in evidence-based medicine. A total of 139 student submissions were evaluated by an experienced faculty member (reference standard) and by the LLM under two prompting strategies: (i) rubric-only and (ii) critical expert-style. Scores across 12 rubric items (maximum 95 points) were compared using paired t -tests, effect sizes, and Bland-Altman analysis. Expert scoring yielded a mean total of 62.0 (SD 13.6). The rubric-only prompt systematically overestimated performance (mean 87.7, SD 8.0, P < 0.001; bias –25.7). The critical prompt produced lower scores (mean 53.5, SD 10.4, P < 0.001; bias +8.5). At the item level, rubric-only prompting aligned better with mechanical tasks (e.g., keywords and referencing), whereas the critical prompt penalized appraisal and synthesis disproportionately. Prompting strategy substantially influenced LLM scoring, generating opposite biases relative to expert evaluation. The novel contribution of this study is that prompt strategy can alter not only the magnitude but also the direction of scoring bias. Calibration approaches, such as context engineering, may help align AI scoring with expert judgment. While AI-generated feedback shows promise for formative assessment, reliable summative use requires careful validation.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Medical Education and Informatics, Faculty of Medicine, Gazi University, Ankara, Turkiye.
- Department of Midwifery, Faculty of Health Sciences, Lokman Hekim University, Ankara, Turkiye.
- Graduate School of Health Sciences, Gazi University, Ankara, Turkiye.
ORCID-Profile: Yavuz Selim Kıyak
Zitieren
APA
Kıyak, Y. S., Bulut, Ö. Ü., Coşkun, Ö., & Budakoğlu, I. İ. (2026). Automated scoring of student videos in medical education: a comparison between a large language model and expert evaluation. Journal of Microbiology & Biology Education, 27(2). https://doi.org/10.1128/jmbe.00010-26
Vancouver
Kıyak YS, Bulut ÖÜ, Coşkun Ö, Budakoğlu Iİ. Automated scoring of student videos in medical education: a comparison between a large language model and expert evaluation. Journal of Microbiology & Biology Education. 2026;27(2). doi:10.1128/jmbe.00010-26
BibTeX
@article{Kyak2026,
author = {Kıyak, Yavuz Selim and Bulut, Özlem Ülkü and Coşkun, Özlem and Budakoğlu, Işıl İrem},
title = {Automated scoring of student videos in medical education: a comparison between a large language model and expert evaluation},
journal = {Journal of Microbiology & Biology Education},
year = {2026},
volume = {27},
number = {2},
doi = {10.1128/jmbe.00010-26}
}
EUVIMED