EUVIMEDEUVIMED entdecken
Open Access

Capability of GPT-4V(ision) in the Japanese National Medical Licensing Examination: Evaluation Study

Takahiro Nakao iD, Soichiro Miki iD, Yuta Nakamura iD, Tomohiro Kikuchi iD, Yukihiro Nomura iD, Shouhei Hanaoka iD, Takeharu Yoshikawa iD, Osamu Abe iD

JMIR Medical Education · 2024

DOI: 10.2196/54393 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Education, 2024. Verfasst von Takahiro Nakao und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Bunkyo-ku, Tokyo, Japan. Hauptthemen laut MEDLINE: Licensure, Medicine. Die Arbeit wurde bisher 45-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Previous research applying large language models (LLMs) to medicine was focused on text-based information. Recently, multimodal variants of LLMs acquired the capability of recognizing images.

Objective

We aim to evaluate the image recognition capability of generative pretrained transformer (GPT)-4V, a recent multimodal LLM developed by OpenAI, in the medical field by testing how visual information affects its performance to answer questions in the 117th Japanese National Medical Licensing Examination.

Methods

We focused on 108 questions that had 1 or more images as part of a question and presented GPT-4V with the same questions under two conditions: (1) with both the question text and associated images and (2) with the question text only. We then compared the difference in accuracy between the 2 conditions using the exact McNemar test.

Results

Among the 108 questions with images, GPT-4V’s accuracy was 68% (73/108) when presented with images and 72% (78/108) when presented without images (P=.36). For the 2 question categories, clinical and general, the accuracies with and those without images were 71% (70/98) versus 78% (76/98; P=.21) and 30% (3/10) versus 20% (2/10; P≥.99), respectively.

Conclusions

The additional information from the images did not significantly improve the performance of GPT-4V in the Japanese National Medical Licensing Examination.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

LicensureMedicineJapanLanguage

Stichwörter der Autor:innen

AIChatGPTGPT-4GPT-4VLLMNLPansweranswersartificial intelligencechatbotchatbotsconversational agentconversational agentsexamexamination

Autor:innen und Institutionen

  • Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Bunkyo-ku, Tokyo, Japan.
  • Department of Radiology, School of Medicine, Jichi Medical University, Shimotsuke, Tochigi, Japan.
  • Center for Frontier Medical Engineering, Chiba University, Inage-ku, Chiba, Japan.
  • Department of Radiology, The University of Tokyo Hospital, Bunkyo-ku, Tokyo, Japan.

ORCID-Profile: Takahiro Nakao, Soichiro Miki, Yuta Nakamura, Tomohiro Kikuchi, Yukihiro Nomura, Shouhei Hanaoka, Takeharu Yoshikawa, Osamu Abe

Zitieren

APA

Nakao, T., Miki, S., Nakamura, Y., Kikuchi, T., Nomura, Y., Hanaoka, S., Yoshikawa, T., & Abe, O. (2024). Capability of GPT-4V(ision) in the Japanese National Medical Licensing Examination: Evaluation Study. JMIR Medical Education. https://doi.org/10.2196/54393

Vancouver

Nakao T, Miki S, Nakamura Y, Kikuchi T, Nomura Y, Hanaoka S, et al. Capability of GPT-4V(ision) in the Japanese National Medical Licensing Examination: Evaluation Study. JMIR Medical Education. 2024. doi:10.2196/54393

BibTeX
@article{Nakao2024,
  author = {Nakao, Takahiro and Miki, Soichiro and Nakamura, Yuta and Kikuchi, Tomohiro and Nomura, Yukihiro and Hanaoka, Shouhei and Yoshikawa, Takeharu and Abe, Osamu},
  title = {Capability of GPT-4V(ision) in the Japanese National Medical Licensing Examination: Evaluation Study},
  journal = {JMIR Medical Education},
  year = {2024},
  doi = {10.2196/54393}
}
Kostenlos starten →