EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Can Large Language Models Generate High-Quality Short-Answer Assessments? A Comparative Study in Undergraduate Medical Education

Leo Morjaria, Levi Burns, Bhavya Gandhi, Keyna Bracken, Muhammad S. Farooq, Anthony J. Levinson, Quang Ngo, Matthew Sibbald

Applied Sciences · 06.03.2026 · 16 (5) · 2535

DOI: 10.3390/app16052535 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 06.03.2026. Verfasst von Leo Morjaria und 7 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: McMaster Education Research, Innovation and Theory.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Generative artificial intelligence (AI) tools including ChatGPT have the potential to augment the process of designing examinations and assessments for medical learners, leading to time and resource savings, and the ability to produce large volumes of practice problems tailored to learner-specific strengths and weaknesses.

Methods

This study compares the quality of free-text assessment problems and answer keys generated by ChatGPT to those produced by faculty educators for a renal and hematology curriculum subunit. Five expert reviewers reviewed a collection of 21 free-text assessment problems, 9 from a collection of historical assessment problems used in an undergraduate medical program and 12 produced with ChatGPT. Reviewers assigned a score from 1 to 5, reflecting the overall quality.

Results

The average quality of problems generated by ChatGPT was greater than that of human-generated problems (4.00 vs. 2.71, p < 0.001). Using ordinal mixed-effect modeling, human-generated problems had significantly lower odds of receiving higher ratings than ChatGPT-generated problems (β = −2.43, 95% confidence interval −3.34 to −1.51, p < 0.001).

Conclusions

It is suggested that ChatGPT can assist expert faculty educators in producing assessment tools, with direct benefits to medical learners, although it cannot entirely replace this role in its current state.

Förderung

  • McMaster Education Research, Innovation and Theory

Zitieren

APA

Morjaria, L., Burns, L., Gandhi, B., Bracken, K., Farooq, M. S., Levinson, A. J., Ngo, Q., & Sibbald, M. (2026). Can Large Language Models Generate High-Quality Short-Answer Assessments? A Comparative Study in Undergraduate Medical Education. Applied Sciences, 16(5), 2535. https://doi.org/10.3390/app16052535

Vancouver

Morjaria L, Burns L, Gandhi B, Bracken K, Farooq MS, Levinson AJ, et al. Can Large Language Models Generate High-Quality Short-Answer Assessments? A Comparative Study in Undergraduate Medical Education. Applied Sciences. 2026;16(5):2535. doi:10.3390/app16052535

BibTeX
@article{Morjaria2026,
  author = {Morjaria, Leo and Burns, Levi and Gandhi, Bhavya and Bracken, Keyna and Farooq, Muhammad S. and Levinson, Anthony J. and Ngo, Quang and Sibbald, Matthew},
  title = {Can Large Language Models Generate High-Quality Short-Answer Assessments? A Comparative Study in Undergraduate Medical Education},
  journal = {Applied Sciences},
  year = {2026},
  volume = {16},
  number = {5},
  pages = {2535},
  doi = {10.3390/app16052535}
}
Kostenlos starten →