Can Large Language Models Generate High-Quality Short-Answer Assessments? A Comparative Study in Undergraduate Medical Education
Applied Sciences · 06.03.2026 · 16 (5) · 2535
DOI: 10.3390/app16052535 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 06.03.2026. Verfasst von Leo Morjaria und 7 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: McMaster Education Research, Innovation and Theory.
Abstract (Originalsprache)
Background
Generative artificial intelligence (AI) tools including ChatGPT have the potential to augment the process of designing examinations and assessments for medical learners, leading to time and resource savings, and the ability to produce large volumes of practice problems tailored to learner-specific strengths and weaknesses.
Methods
This study compares the quality of free-text assessment problems and answer keys generated by ChatGPT to those produced by faculty educators for a renal and hematology curriculum subunit. Five expert reviewers reviewed a collection of 21 free-text assessment problems, 9 from a collection of historical assessment problems used in an undergraduate medical program and 12 produced with ChatGPT. Reviewers assigned a score from 1 to 5, reflecting the overall quality.
Results
The average quality of problems generated by ChatGPT was greater than that of human-generated problems (4.00 vs. 2.71, p < 0.001). Using ordinal mixed-effect modeling, human-generated problems had significantly lower odds of receiving higher ratings than ChatGPT-generated problems (β = −2.43, 95% confidence interval −3.34 to −1.51, p < 0.001).
Conclusions
It is suggested that ChatGPT can assist expert faculty educators in producing assessment tools, with direct benefits to medical learners, although it cannot entirely replace this role in its current state.
Förderung
- McMaster Education Research, Innovation and Theory
Zitieren
APA
Morjaria, L., Burns, L., Gandhi, B., Bracken, K., Farooq, M. S., Levinson, A. J., Ngo, Q., & Sibbald, M. (2026). Can Large Language Models Generate High-Quality Short-Answer Assessments? A Comparative Study in Undergraduate Medical Education. Applied Sciences, 16(5), 2535. https://doi.org/10.3390/app16052535
Vancouver
Morjaria L, Burns L, Gandhi B, Bracken K, Farooq MS, Levinson AJ, et al. Can Large Language Models Generate High-Quality Short-Answer Assessments? A Comparative Study in Undergraduate Medical Education. Applied Sciences. 2026;16(5):2535. doi:10.3390/app16052535
BibTeX
@article{Morjaria2026,
author = {Morjaria, Leo and Burns, Levi and Gandhi, Bhavya and Bracken, Keyna and Farooq, Muhammad S. and Levinson, Anthony J. and Ngo, Quang and Sibbald, Matthew},
title = {Can Large Language Models Generate High-Quality Short-Answer Assessments? A Comparative Study in Undergraduate Medical Education},
journal = {Applied Sciences},
year = {2026},
volume = {16},
number = {5},
pages = {2535},
doi = {10.3390/app16052535}
}
EUVIMED