EUVIMEDEUVIMED entdecken
Open Access

Performance of large language models at the MRCS Part A: a tool for medical education?

A Yiu, K Lam

The Annals of The Royal College of Surgeons of England · 2025

DOI: 10.1308/rcsann.2023.0085 ↗

Kurz gefasst

Fachartikel in der Zeitschrift The Annals of The Royal College of Surgeons of England, 2025. Verfasst von A Yiu und 1 weiteren Autor:innen. Beteiligte Einrichtung: King's College Hospital NHS Foundation Trust, UK. Hauptthemen laut MEDLINE: Educational Measurement, Education, Medical, General Surgery, Language. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 18 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

INTRODUCTION: The Intercollegiate Membership of the Royal College of Surgeons examination (MRCS) Part A assesses generic surgical sciences and applied knowledge using 300 multiple-choice Single Best Answer items. Large Language Models (LLMs) are trained on vast amounts of text to generate natural language outputs, and applications in healthcare and medical education are rising. METHODS: Two LLMs, ChatGPT (OpenAI) and Bard (Google AI), were tested using 300 questions from a popular MRCS Part A question bank without/with need for justification (NJ/J). LLM outputs were scored according to accuracy, concordance and insight. RESULTS: ChatGPT achieved 85.7%/84.3% accuracy for NJ/J encodings. Bard achieved 64%/64.3% accuracy for NJ/J encodings. ChatGPT and Bard displayed high levels of concordance for NJ (95.3%; 81.7%) and J (93.7%; 79.7%) encodings, respectively. ChatGPT and Bard provided an insightful statement in >98% and >86% outputs, respectively. DISCUSSION: This study demonstrates that ChatGPT achieves passing-level accuracy at MRCS Part A, and both LLMs achieve high concordance and provide insightful responses to test questions. Instances of clinically inappropriate or inaccurate decision-making, incomplete appreciation of nuanced clinical scenarios and utilisation of out-of-date guidance was, however, noted. LLMs are accessible and time-efficient tools, access vast clinical knowledge, and may reduce the emphasis on factual recall in medical education and assessment. CONCLUSION: ChatGPT achieves passing-level accuracy for MRCS Part A with concordant and insightful outputs. Future applications of LLMs in healthcare must be cautious of hallucinations and incorrect reasoning but have the potential to develop AI-supported clinicians.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Educational MeasurementEducation, MedicalGeneral SurgeryLanguageHumansLarge Language Models

Stichwörter der Autor:innen

BardChatGPTEducationExaminationLarge language modelMRCS

Autor:innen und Institutionen

  • King's College Hospital NHS Foundation Trust, UK.
  • Imperial College London, UK.

Zitieren

APA

Yiu, A., & Lam, K. (2025). Performance of large language models at the MRCS Part A: a tool for medical education?. The Annals of The Royal College of Surgeons of England. https://doi.org/10.1308/rcsann.2023.0085

Vancouver

Yiu A, Lam K. Performance of large language models at the MRCS Part A: a tool for medical education?. The Annals of The Royal College of Surgeons of England. 2025. doi:10.1308/rcsann.2023.0085

BibTeX
@article{Yiu2025,
  author = {Yiu, A and Lam, K},
  title = {Performance of large language models at the MRCS Part A: a tool for medical education?},
  journal = {The Annals of The Royal College of Surgeons of England},
  year = {2025},
  doi = {10.1308/rcsann.2023.0085}
}
Kostenlos starten →