EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Can ChatGPT pass the MRCP (UK) written examinations? Analysis of performance and errors using a clinical decision-reasoning framework

Amy Maitland iD, Ross Fowkes, Stuart Maitland

BMJ Open · 2024

DOI: 10.1136/bmjopen-2023-080558 ↗

Kurz gefasst

Fachartikel in der Zeitschrift BMJ Open, 2024. Verfasst von Amy Maitland und 2 weiteren Autor:innen. Beteiligte Einrichtung: Health Education England North East, Newcastle upon Tyne, UK. Hauptthemen laut MEDLINE: Cholangiopancreatography, Magnetic Resonance, Clinical Reasoning. Die Arbeit wurde bisher 51-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: NIHR Newcastle Biomedical Research Centre, National Institute for Health and Care Research.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Large language models (LLMs) such as ChatGPT are being developed for use in research, medical education and clinical decision systems. However, as their usage increases, LLMs face ongoing regulatory concerns. This study aims to analyse ChatGPT’s performance on a postgraduate examination to identify areas of strength and weakness, which may provide further insight into their role in healthcare.

Design

We evaluated the performance of ChatGPT 4 (24 May 2023 version) on official MRCP (Membership of the Royal College of Physicians) parts 1 and 2 written examination practice questions. Statistical analysis was performed using Python. Spearman rank correlation assessed the relationship between the probability of correctly answering a question and two variables: question difficulty and question length. Incorrectly answered questions were analysed further using a clinical reasoning framework to assess the errors made.

Setting

Online using ChatGPT web interface. Primary and secondary outcome measures Primary outcome was the score (percentage questions correct) in the MRCP postgraduate written examinations. Secondary outcomes were qualitative categorisation of errors using a clinical decision-making framework.

Results

ChatGPT achieved accuracy rates of 86.3% (part 1) and 70.3% (part 2). Weak but significant correlations were found between ChatGPT’s accuracy and both just-passing rates in part 2 (r=0.34, p=0.0001) and question length in part 1 (r=−0.19, p=0.008). Eight types of error were identified, with the most frequent being factual errors, context errors and omission errors.

Conclusion

ChatGPT performance greatly exceeded the passing mark for both exams. Multiple choice examinations provide a benchmark for LLM performance which is comparable to human demonstrations of knowledge, while also highlighting the errors LLMs make. Understanding the reasons behind ChatGPT’s errors allows us to develop strategies to prevent them in medical devices that incorporate LLM technology.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Cholangiopancreatography, Magnetic ResonanceClinical ReasoningHumansClinical Decision-MakingBenchmarkingUnited Kingdom

Stichwörter der Autor:innen

Clinical Decision-MakingHealth informaticsMEDICAL EDUCATION & TRAININGQUALITATIVE RESEARCH

Autor:innen und Institutionen

  • Health Education England North East, Newcastle upon Tyne, UK.
  • The Newcastle Upon Tyne NHS Hospitals Foundation Trust, Newcastle upon Tyne, UK stu.maitland@newcastle.ac.uk.
  • Newcastle University Translational and Clinical Research Institute, Newcastle upon Tyne, UK.

ORCID-Profile: Amy Maitland

Förderung

  • NIHR Newcastle Biomedical Research Centre
  • National Institute for Health and Care Research

Zitieren

APA

Maitland, A., Fowkes, R., & Maitland, S. (2024). Can ChatGPT pass the MRCP (UK) written examinations? Analysis of performance and errors using a clinical decision-reasoning framework. BMJ Open. https://doi.org/10.1136/bmjopen-2023-080558

Vancouver

Maitland A, Fowkes R, Maitland S. Can ChatGPT pass the MRCP (UK) written examinations? Analysis of performance and errors using a clinical decision-reasoning framework. BMJ Open. 2024. doi:10.1136/bmjopen-2023-080558

BibTeX
@article{Maitland2024,
  author = {Maitland, Amy and Fowkes, Ross and Maitland, Stuart},
  title = {Can ChatGPT pass the MRCP (UK) written examinations? Analysis of performance and errors using a clinical decision-reasoning framework},
  journal = {BMJ Open},
  year = {2024},
  doi = {10.1136/bmjopen-2023-080558}
}
Kostenlos starten →