Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study
JMIR Medical Education · 29.06.2023 · 9 · e48002
DOI: 10.2196/48002 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Medical Education, online erschienen am 29.06.2023. Verfasst von Soshi Takagi und 3 weiteren Autor:innen. Beteiligte Einrichtung: Faculty of Medicine, Shimane University, Izumo, Japan. Die Arbeit wurde bisher 287-mal zitiert (Crossref) und nennt 26 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
The competence of ChatGPT (Chat Generative Pre-Trained Transformer) in non-English languages is not well studied.
Objective
This study compared the performances of GPT-3.5 (Generative Pre-trained Transformer) and GPT-4 on the Japanese Medical Licensing Examination (JMLE) to evaluate the reliability of these models for clinical reasoning and medical knowledge in non-English languages.
Methods
This study used the default mode of ChatGPT, which is based on GPT-3.5; the GPT-4 model of ChatGPT Plus; and the 117th JMLE in 2023. A total of 254 questions were included in the final analysis, which were categorized into 3 types, namely general, clinical, and clinical sentence questions.
Results
The results indicated that GPT-4 outperformed GPT-3.5 in terms of accuracy, particularly for general, clinical, and clinical sentence questions. GPT-4 also performed better on difficult questions and specific disease questions. Furthermore, GPT-4 achieved the passing criteria for the JMLE, indicating its reliability for clinical reasoning and medical knowledge in non-English languages.
Conclusions
GPT-4 could become a valuable tool for medical education and clinical support in non–English-speaking regions, such as Japan.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Faculty of Medicine, Shimane University, Izumo, Japan.
- General Medicine Center, Shimane University Hospital, Izumo, Japan.
- Department of Internal Medicine, University of Michigan Medical School, Ann Arbor, MI, United States.
- Medicine Service, VA Ann Arbor Healthcare System, Ann Arbor, MI, United States.
ORCID-Profile: Soshi Takagi, Takashi Watari, Ayano Erabi, Kota Sakaguchi
Zitieren
APA
Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K. (2023). Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study. JMIR Medical Education, 9, e48002. https://doi.org/10.2196/48002
Vancouver
Takagi S, Watari T, Erabi A, Sakaguchi K. Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study. JMIR Medical Education. 2023;9:e48002. doi:10.2196/48002
BibTeX
@article{Takagi2023,
author = {Takagi, Soshi and Watari, Takashi and Erabi, Ayano and Sakaguchi, Kota},
title = {Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study},
journal = {JMIR Medical Education},
year = {2023},
volume = {9},
pages = {e48002},
doi = {10.2196/48002}
}
EUVIMED