EUVIMEDEUVIMED entdecken
BeobachtungsstudieOpen Access

Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study

Annika Meyer iD, Janik Riese iD, Thomas Streichert iD

JMIR Medical Education · 08.02.2024 · 10 · e50965

DOI: 10.2196/50965 ↗

Kurz gefasst

Beobachtungsstudie in der Zeitschrift JMIR Medical Education, online erschienen am 08.02.2024. Verfasst von Annika Meyer und 2 weiteren Autor:innen. Beteiligte Einrichtung: Institute for Clinical Chemistry, University Hospital Cologne, Cologne, Germany. Hauptthemen laut MEDLINE: Artificial Intelligence, Education, Medical, Students, Medical, Educational Measurement. Die Arbeit wurde bisher 120-mal zitiert (Crossref) und nennt 59 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The potential of artificial intelligence (AI)–based large language models, such as ChatGPT, has gained significant attention in the medical field. This enthusiasm is driven not only by recent breakthroughs and improved accessibility, but also by the prospect of democratizing medical knowledge and promoting equitable health care. However, the performance of ChatGPT is substantially influenced by the input language, and given the growing public trust in this AI tool compared to that in traditional sources of information, investigating its medical accuracy across different languages is of particular importance.

Objective

This study aimed to compare the performance of GPT-3.5 and GPT-4 with that of medical students on the written German medical licensing examination.

Methods

To assess GPT-3.5’s and GPT-4's medical proficiency, we used 937 original multiple-choice questions from 3 written German medical licensing examinations in October 2021, April 2022, and October 2022.

Results

GPT-4 achieved an average score of 85% and ranked in the 92.8th, 99.5th, and 92.6th percentiles among medical students who took the same examinations in October 2021, April 2022, and October 2022, respectively. This represents a substantial improvement of 27% compared to GPT-3.5, which only passed 1 out of the 3 examinations. While GPT-3.5 performed well in psychiatry questions, GPT-4 exhibited strengths in internal medicine and surgery but showed weakness in academic research.

Conclusions

The study results highlight ChatGPT’s remarkable improvement from moderate (GPT-3.5) to high competency (GPT-4) in answering medical licensing examination questions in German. While GPT-4’s predecessor (GPT-3.5) was imprecise and inconsistent, it demonstrates considerable potential to improve medical education and patient care, provided that medically trained users critically evaluate its results. As the replacement of search engines by AI tools seems possible in the future, further studies with nonprofessional questions are needed to assess the safety and accuracy of ChatGPT for the general population.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceEducation, MedicalStudents, MedicalEducational MeasurementHumansLanguage

Stichwörter der Autor:innen

ChatGPTLLMartificial intelligencegeneral populationimprovementlarge language modellicensing examlicensing examinationlicensure examinationmedical accuracymedical educationmedical examinationsmedical examspatient carepublic trust

Autor:innen und Institutionen

  • Institute for Clinical Chemistry, University Hospital Cologne, Cologne, Germany.
  • Department of General Surgery, Visceral, Thoracic and Vascular Surgery, University Hospital Greifswald, Greifswald, Germany.

ORCID-Profile: Annika Meyer, Janik Riese, Thomas Streichert

Zitieren

APA

Meyer, A., Riese, J., & Streichert, T. (2024). Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study. JMIR Medical Education, 10, e50965. https://doi.org/10.2196/50965

Vancouver

Meyer A, Riese J, Streichert T. Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study. JMIR Medical Education. 2024;10:e50965. doi:10.2196/50965

BibTeX
@article{Meyer2024,
  author = {Meyer, Annika and Riese, Janik and Streichert, Thomas},
  title = {Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study},
  journal = {JMIR Medical Education},
  year = {2024},
  volume = {10},
  pages = {e50965},
  doi = {10.2196/50965}
}
Kostenlos starten →