Revolution or risk?—Assessing the potential and challenges of GPT-4V in radiologic image interpretation
European Radiology · 18.10.2024 · 35 (3) · 1111-1121
DOI: 10.1007/s00330-024-11115-6 ↗Kurz gefasst
Fachartikel in der Zeitschrift European Radiology, online erschienen am 18.10.2024. Verfasst von Marc Sebastian Huppertz und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany. mhuppertz@ukaachen.de. Hauptthemen laut MEDLINE: Image Interpretation, Computer-Assisted. Die Arbeit wurde bisher 65-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: HORIZON EUROPE European Research Council, Deutsche Forschungsgemeinschaft, Bundesministerium für Bildung und Forschung.
Abstract (Originalsprache)
Objectives
ChatGPT-4 Vision (GPT-4V) is a state-of-the-art multimodal large language model (LLM) that may be queried using images. We aimed to evaluate the tool’s diagnostic performance when autonomously assessing clinical imaging studies. Materials and methods A total of 206 imaging studies (i.e., radiography ( n = 60), CT ( n = 60), MRI ( n = 60), and angiography ( n = 26)) with unequivocal findings and established reference diagnoses from the radiologic practice of a large university hospital were accessed. Readings were performed uncontextualized, with only the image provided, and contextualized, with additional clinical and demographic information. Responses were assessed along multiple diagnostic dimensions and analyzed using appropriate statistical tests.
Results
With its pronounced propensity to favor context over image information, the tool’s diagnostic accuracy improved from 8.3% (uncontextualized) to 29.1% (contextualized, first diagnosis correct) and 63.6% (contextualized, correct diagnosis among differential diagnoses) ( p ≤ 0.001, Cochran’s Q test). Diagnostic accuracy declined by up to 30% when 20 images were re-read after 30 and 90 days and seemed unrelated to the tool’s self-reported confidence (Spearman’s ρ = 0.117 ( p = 0.776)). While the described imaging findings matched the suggested diagnoses in 92.7%, indicating valid diagnostic reasoning, the tool fabricated 258 imaging findings in 412 responses and misidentified imaging modalities or anatomic regions in 65 images.
Conclusion
GPT-4V, in its current form, cannot reliably interpret radiologic images. Its tendency to disregard the image, fabricate findings, and misidentify details, especially without clinical context, may misguide healthcare providers and put patients at risk. Key Points Question Can Generative Pre-trained Transformer 4 Vision (GPT-4V) interpret radiologic images—with and without clinical context?
Findings
GPT-4V performed poorly, demonstrating diagnostic accuracy rates of 8% (uncontextualized), 29% (contextualized, most likely diagnosis correct), and 64% (contextualized, correct diagnosis among differential diagnoses). Clinical relevance The utility of commercial multimodal large language models, such as GPT-4V, in radiologic practice is limited. Without clinical context, diagnostic errors and fabricated findings may compromise patient safety and misguide clinical decision-making. These models must be further refined to be beneficial.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany. mhuppertz@ukaachen.de.
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany.
- Department of Diagnostic and Interventional Neuroradiology, University Hospital RWTH Aachen, Aachen, Germany.
Förderung
- HORIZON EUROPE European Research Council
- Deutsche Forschungsgemeinschaft
- Bundesministerium für Bildung und Forschung
Zitieren
APA
Huppertz, M. S., Siepmann, R., Topp, D., Nikoubashman, O., Yüksel, C., Kuhl, C. K., Truhn, D., & Nebelung, S. (2024). Revolution or risk?—Assessing the potential and challenges of GPT-4V in radiologic image interpretation. European Radiology, 35(3), 1111-1121. https://doi.org/10.1007/s00330-024-11115-6
Vancouver
Huppertz MS, Siepmann R, Topp D, Nikoubashman O, Yüksel C, Kuhl CK, et al. Revolution or risk?—Assessing the potential and challenges of GPT-4V in radiologic image interpretation. European Radiology. 2024;35(3):1111-1121. doi:10.1007/s00330-024-11115-6
BibTeX
@article{Huppertz2024,
author = {Huppertz, Marc Sebastian and Siepmann, Robert and Topp, David and Nikoubashman, Omid and Yüksel, Can and Kuhl, Christiane Katharina and Truhn, Daniel and Nebelung, Sven},
title = {Revolution or risk?—Assessing the potential and challenges of GPT-4V in radiologic image interpretation},
journal = {European Radiology},
year = {2024},
volume = {35},
number = {3},
pages = {1111--1121},
doi = {10.1007/s00330-024-11115-6}
}
EUVIMED