Performance of large language models at the MRCS Part A: a tool for medical education?
The Annals of The Royal College of Surgeons of England · 2025
DOI: 10.1308/rcsann.2023.0085 ↗Kurz gefasst
Fachartikel in der Zeitschrift The Annals of The Royal College of Surgeons of England, 2025. Verfasst von A Yiu und 1 weiteren Autor:innen. Beteiligte Einrichtung: King's College Hospital NHS Foundation Trust, UK. Hauptthemen laut MEDLINE: Educational Measurement, Education, Medical, General Surgery, Language. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 18 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
INTRODUCTION: The Intercollegiate Membership of the Royal College of Surgeons examination (MRCS) Part A assesses generic surgical sciences and applied knowledge using 300 multiple-choice Single Best Answer items. Large Language Models (LLMs) are trained on vast amounts of text to generate natural language outputs, and applications in healthcare and medical education are rising. METHODS: Two LLMs, ChatGPT (OpenAI) and Bard (Google AI), were tested using 300 questions from a popular MRCS Part A question bank without/with need for justification (NJ/J). LLM outputs were scored according to accuracy, concordance and insight. RESULTS: ChatGPT achieved 85.7%/84.3% accuracy for NJ/J encodings. Bard achieved 64%/64.3% accuracy for NJ/J encodings. ChatGPT and Bard displayed high levels of concordance for NJ (95.3%; 81.7%) and J (93.7%; 79.7%) encodings, respectively. ChatGPT and Bard provided an insightful statement in >98% and >86% outputs, respectively. DISCUSSION: This study demonstrates that ChatGPT achieves passing-level accuracy at MRCS Part A, and both LLMs achieve high concordance and provide insightful responses to test questions. Instances of clinically inappropriate or inaccurate decision-making, incomplete appreciation of nuanced clinical scenarios and utilisation of out-of-date guidance was, however, noted. LLMs are accessible and time-efficient tools, access vast clinical knowledge, and may reduce the emphasis on factual recall in medical education and assessment. CONCLUSION: ChatGPT achieves passing-level accuracy for MRCS Part A with concordant and insightful outputs. Future applications of LLMs in healthcare must be cautious of hallucinations and incorrect reasoning but have the potential to develop AI-supported clinicians.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- King's College Hospital NHS Foundation Trust, UK.
- Imperial College London, UK.
Zitieren
APA
Yiu, A., & Lam, K. (2025). Performance of large language models at the MRCS Part A: a tool for medical education?. The Annals of The Royal College of Surgeons of England. https://doi.org/10.1308/rcsann.2023.0085
Vancouver
Yiu A, Lam K. Performance of large language models at the MRCS Part A: a tool for medical education?. The Annals of The Royal College of Surgeons of England. 2025. doi:10.1308/rcsann.2023.0085
BibTeX
@article{Yiu2025,
author = {Yiu, A and Lam, K},
title = {Performance of large language models at the MRCS Part A: a tool for medical education?},
journal = {The Annals of The Royal College of Surgeons of England},
year = {2025},
doi = {10.1308/rcsann.2023.0085}
}
EUVIMED