EUVIMEDEUVIMED entdecken
Open Access

Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study

Soshi Takagi iD, Takashi Watari iD, Ayano Erabi iD, Kota Sakaguchi iD

JMIR Medical Education · 29.06.2023 · 9 · e48002

DOI: 10.2196/48002 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Education, online erschienen am 29.06.2023. Verfasst von Soshi Takagi und 3 weiteren Autor:innen. Beteiligte Einrichtung: Faculty of Medicine, Shimane University, Izumo, Japan. Die Arbeit wurde bisher 287-mal zitiert (Crossref) und nennt 26 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The competence of ChatGPT (Chat Generative Pre-Trained Transformer) in non-English languages is not well studied.

Objective

This study compared the performances of GPT-3.5 (Generative Pre-trained Transformer) and GPT-4 on the Japanese Medical Licensing Examination (JMLE) to evaluate the reliability of these models for clinical reasoning and medical knowledge in non-English languages.

Methods

This study used the default mode of ChatGPT, which is based on GPT-3.5; the GPT-4 model of ChatGPT Plus; and the 117th JMLE in 2023. A total of 254 questions were included in the final analysis, which were categorized into 3 types, namely general, clinical, and clinical sentence questions.

Results

The results indicated that GPT-4 outperformed GPT-3.5 in terms of accuracy, particularly for general, clinical, and clinical sentence questions. GPT-4 also performed better on difficult questions and specific disease questions. Furthermore, GPT-4 achieved the passing criteria for the JMLE, indicating its reliability for clinical reasoning and medical knowledge in non-English languages.

Conclusions

GPT-4 could become a valuable tool for medical education and clinical support in non–English-speaking regions, such as Japan.

Schlagwörter

Stichwörter der Autor:innen

AIChat Generative Pre-trained TransformerChatGPTGPT-4Generative Pre-trained Transformer 4Japanese Medical Licensing Examinationartificial intelligenceclinical supportlearning modelmedical educationmedical licensing

Autor:innen und Institutionen

  • Faculty of Medicine, Shimane University, Izumo, Japan.
  • General Medicine Center, Shimane University Hospital, Izumo, Japan.
  • Department of Internal Medicine, University of Michigan Medical School, Ann Arbor, MI, United States.
  • Medicine Service, VA Ann Arbor Healthcare System, Ann Arbor, MI, United States.

ORCID-Profile: Soshi Takagi, Takashi Watari, Ayano Erabi, Kota Sakaguchi

Zitieren

APA

Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K. (2023). Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study. JMIR Medical Education, 9, e48002. https://doi.org/10.2196/48002

Vancouver

Takagi S, Watari T, Erabi A, Sakaguchi K. Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study. JMIR Medical Education. 2023;9:e48002. doi:10.2196/48002

BibTeX
@article{Takagi2023,
  author = {Takagi, Soshi and Watari, Takashi and Erabi, Ayano and Sakaguchi, Kota},
  title = {Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study},
  journal = {JMIR Medical Education},
  year = {2023},
  volume = {9},
  pages = {e48002},
  doi = {10.2196/48002}
}
Kostenlos starten →