The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis
PLOS One · 02.01.2026 · 21 (1) · e0340277
DOI: 10.1371/journal.pone.0340277 ↗Kurz gefasst
Netzwerk-Metaanalyse in der Zeitschrift PLOS One, online erschienen am 02.01.2026. Verfasst von Lauren Riehm und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Anesthesia and Pain Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada. Hauptthemen laut MEDLINE: Education, Medical, Educational Measurement, Health Occupations, Language. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Purpose
Large language models (LLMs) have the potential to change medical education. Whether LLMs can generate multiple-choice questions (MCQs) that are of similar quality to those created by humans is unclear. This investigation assessed the quality of MCQs generated by LLMs compared to humans.
Methods
This review was registered with PROSPERO (CRD42025608775). A systematic review and frequentist random-effects network meta-analysis (NMA) or pairwise meta-analysis was performed. Ovid MEDLINE, Ovid EMBASE, and Scopus were searched from inception to November 1, 2024. The quality of MCQs was assessed with seven pre-defined outcomes: question relevance, clarity, accuracy/correctness; distractor quality; item difficulty analysis; and item discrimination analysis (point biserial correlation and item discrimination index). Continuous data were transformed to a 10-point scale to facilitate statistical analysis and reported as mean differences (MD). The MERSQI and the Grade of Recommendations, Assessment, Development and Evaluation (GRADE) NMA guidelines were used to assess risk of bias and certainty of evidence assessments.
Results
Five LLMs were included. NMA demonstrated that ChatGPT 4 generated similar quality MCQs to humans with regards to question relevance (MD −0.13; 95% CI: −0.44,0.18; GRADE: VERY LOW), question clarity (MD −0.03; 95% CI: −0.15,0.10; GRADE: VERY LOW), and distractor quality (MD −0.10; 95% CI: −0.24,0.04; GRADE: VERY LOW); however, MCQs generated by Llama 2 performed worse than humans with regards to question clarity (MD −1.21; 95% CI: −1.60,-0.82; GRADE: VERY LOW) and distractor quality (MD −1.50; 95% CI: −2.03,-0.97; GRADE: VERY LOW). Exploratory post-hoc t-tests demonstrated that ChatGPT 3.5 performed worse than Llama 2 and ChatGPT 4 with regards to question clarity and distractor quality (p < 0.001).
Conclusion
ChatGPT 4 may create similar quality MCQs to humans, whereas ChatGPT 3.5 and Llama 2 may be of worse quality. Further studies that directly compare these LLMs to human-generated questions and administer MCQs to students are required.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Anesthesia and Pain Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada.
- Department of Anesthesiology and Pain Medicine, University of Toronto, Toronto, Ontario, Canada.
- Division of Ophthalmology, Department of Surgery, McMaster University, Hamilton, Ontario, Canada.
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada.
- Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada.
- Department of Surgery, McMaster University, Hamilton, Ontario, Canada.
ORCID-Profile: Lauren Riehm, Keean Nanji, Wesla Pfeifer
Zitieren
APA
Riehm, L., Nanji, K., Lakhani, M., Pankiv, E., Hasanee, D., & Pfeifer, W. (2026). The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis. PLOS One, 21(1), e0340277. https://doi.org/10.1371/journal.pone.0340277
Vancouver
Riehm L, Nanji K, Lakhani M, Pankiv E, Hasanee D, Pfeifer W. The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis. PLOS One. 2026;21(1):e0340277. doi:10.1371/journal.pone.0340277
BibTeX
@article{Riehm2026,
author = {Riehm, Lauren and Nanji, Keean and Lakhani, Moiz and Pankiv, Evelina and Hasanee, Dean and Pfeifer, Wesla},
title = {The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis},
journal = {PLOS One},
year = {2026},
volume = {21},
number = {1},
pages = {e0340277},
doi = {10.1371/journal.pone.0340277}
}
EUVIMED