ChatGPT’s diagnostic performance based on textual vs. visual information compared to radiologists’ diagnostic performance in musculoskeletal radiology
European Radiology · 12.07.2024 · 35 (1) · 506-516
DOI: 10.1007/s00330-024-10902-5 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift European Radiology, online erschienen am 12.07.2024. Verfasst von Daisuke Horiuchi und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Diagnostic and Interventional Radiology, Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan. Hauptthemen laut MEDLINE: Radiologists, Musculoskeletal Diseases. Die Arbeit wurde bisher 78-mal zitiert (Crossref) und nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Guerbet.
Abstract (Originalsprache)
Objectives
To compare the diagnostic accuracy of Generative Pre-trained Transformer (GPT)-4-based ChatGPT, GPT-4 with vision (GPT-4V) based ChatGPT, and radiologists in musculoskeletal radiology. Materials and methods We included 106 “Test Yourself” cases from Skeletal Radiology between January 2014 and September 2023. We input the medical history and imaging findings into GPT-4-based ChatGPT and the medical history and images into GPT-4V-based ChatGPT, then both generated a diagnosis for each case. Two radiologists (a radiology resident and a board-certified radiologist) independently provided diagnoses for all cases. The diagnostic accuracy rates were determined based on the published ground truth. Chi-square tests were performed to compare the diagnostic accuracy of GPT-4-based ChatGPT, GPT-4V-based ChatGPT, and radiologists.
Results
GPT-4-based ChatGPT significantly outperformed GPT-4V-based ChatGPT ( p < 0.001) with accuracy rates of 43% (46/106) and 8% (9/106), respectively. The radiology resident and the board-certified radiologist achieved accuracy rates of 41% (43/106) and 53% (56/106). The diagnostic accuracy of GPT-4-based ChatGPT was comparable to that of the radiology resident, but was lower than that of the board-certified radiologist although the differences were not significant ( p = 0.78 and 0.22, respectively). The diagnostic accuracy of GPT-4V-based ChatGPT was significantly lower than those of both radiologists ( p < 0.001 and < 0.001, respectively).
Conclusion
GPT-4-based ChatGPT demonstrated significantly higher diagnostic accuracy than GPT-4V-based ChatGPT. While GPT-4-based ChatGPT’s diagnostic performance was comparable to radiology residents, it did not reach the performance level of board-certified radiologists in musculoskeletal radiology. Clinical relevance statement GPT-4-based ChatGPT outperformed GPT-4V-based ChatGPT and was comparable to radiology residents, but it did not reach the level of board-certified radiologists in musculoskeletal radiology. Radiologists should comprehend ChatGPT’s current performance as a diagnostic tool for optimal utilization. Key Points This study compared the diagnostic performance of GPT-4-based ChatGPT, GPT-4V-based ChatGPT, and radiologists in musculoskeletal radiology. GPT-4-based ChatGPT was comparable to radiology residents, but did not reach the level of board-certified radiologists. When utilizing ChatGPT, it is crucial to input appropriate descriptions of imaging findings rather than the images. Graphical Abstract
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Diagnostic and Interventional Radiology, Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan.
- Department of Diagnostic and Interventional Radiology, Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan. ai.labo.ocu@gmail.com.
- Department of Artificial Intelligence, Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan. ai.labo.ocu@gmail.com.
Förderung
- Guerbet
Zitieren
APA
Horiuchi, D., Tatekawa, H., Oura, T., Shimono, T., Walston, S. L., Takita, H., Matsushita, S., Mitsuyama, Y., Miki, Y., & Ueda, D. (2024). ChatGPT’s diagnostic performance based on textual vs. visual information compared to radiologists’ diagnostic performance in musculoskeletal radiology. European Radiology, 35(1), 506-516. https://doi.org/10.1007/s00330-024-10902-5
Vancouver
Horiuchi D, Tatekawa H, Oura T, Shimono T, Walston SL, Takita H, et al. ChatGPT’s diagnostic performance based on textual vs. visual information compared to radiologists’ diagnostic performance in musculoskeletal radiology. European Radiology. 2024;35(1):506-516. doi:10.1007/s00330-024-10902-5
BibTeX
@article{Horiuchi2024,
author = {Horiuchi, Daisuke and Tatekawa, Hiroyuki and Oura, Tatsushi and Shimono, Taro and Walston, Shannon L and Takita, Hirotaka and Matsushita, Shu and Mitsuyama, Yasuhito and Miki, Yukio and Ueda, Daiju},
title = {ChatGPT’s diagnostic performance based on textual vs. visual information compared to radiologists’ diagnostic performance in musculoskeletal radiology},
journal = {European Radiology},
year = {2024},
volume = {35},
number = {1},
pages = {506--516},
doi = {10.1007/s00330-024-10902-5}
}
EUVIMED