EUVIMEDEUVIMED entdecken
Open Access

The performance of large language models in intercollegiate Membership of the Royal College of Surgeons examination

J Chan, T Dong, G D Angelini

The Annals of The Royal College of Surgeons of England · 2024

DOI: 10.1308/rcsann.2024.0023 ↗

Kurz gefasst

Fachartikel in der Zeitschrift The Annals of The Royal College of Surgeons of England, 2024. Verfasst von J Chan und 2 weiteren Autor:innen. Beteiligte Einrichtung: Bristol Heart Institute, University of Bristol, UK. Hauptthemen laut MEDLINE: Educational Measurement. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 7 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

INTRODUCTION: Large language models (LLM), such as Chat Generative Pre-trained Transformer (ChatGPT) and Bard utilise deep learning algorithms that have been trained on a massive data set of text and code to generate human-like responses. Several studies have demonstrated satisfactory performance on postgraduate examinations, including the United States Medical Licensing Examination. We aimed to evaluate artificial intelligence performance in Part A of the intercollegiate Membership of the Royal College of Surgeons (MRCS) examination. METHODS: The MRCS mock examination from Pastest, a commonly used question bank for examinees, was used to assess the performance of three LLMs: GPT-3.5, GPT 4.0 and Bard. Three hundred mock questions were input into the three LLMs, and the responses provided by the LLMs were recorded and analysed. The pass mark was set at 70%. RESULTS: The overall accuracies for GPT-3.5, GPT 4.0 and Bard were 67.33%, 71.67% and 65.67%, respectively (p = 0.27). The performances of GPT-3.5, GPT 4.0 and Bard in Applied Basic Sciences were 68.89%, 72.78% and 63.33% (p = 0.15), respectively. Furthermore, the three LLMs obtained correct answers in 65.00%, 70.00% and 69.17% of the Principles of Surgery in General questions (p = 0.67). There were no differences in performance in the overall and subcategories among the three LLMs. CONCLUSIONS: Our findings demonstrated satisfactory performance for all three LLMs in the MRCS Part A examination, with GPT 4.0 the only LLM that achieved the pass mark set.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Educational MeasurementHumansArtificial IntelligenceSurgeonsGeneral SurgeryClinical CompetenceDeep LearningSocieties, MedicalUnited Kingdom

Stichwörter der Autor:innen

BardChatGPTLarge language modelsMRCS

Autor:innen und Institutionen

  • Bristol Heart Institute, University of Bristol, UK.

Zitieren

APA

Chan, J., Dong, T., & Angelini, G. D. (2024). The performance of large language models in intercollegiate Membership of the Royal College of Surgeons examination. The Annals of The Royal College of Surgeons of England. https://doi.org/10.1308/rcsann.2024.0023

Vancouver

Chan J, Dong T, Angelini GD. The performance of large language models in intercollegiate Membership of the Royal College of Surgeons examination. The Annals of The Royal College of Surgeons of England. 2024. doi:10.1308/rcsann.2024.0023

BibTeX
@article{Chan2024,
  author = {Chan, J and Dong, T and Angelini, G D},
  title = {The performance of large language models in intercollegiate Membership of the Royal College of Surgeons examination},
  journal = {The Annals of The Royal College of Surgeons of England},
  year = {2024},
  doi = {10.1308/rcsann.2024.0023}
}
Kostenlos starten →