New frontiers in radiologic interpretation: evaluating the effectiveness of large language models in pneumothorax diagnosis
PLOS One · 2025
DOI: 10.1371/journal.pone.0331962 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS One, 2025. Verfasst von Bensu Bulut und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Emergency Medicine, Ankara Gulhane Training and Research Hospital, Health Science University, Ankara, Turkey. Hauptthemen laut MEDLINE: Pneumothorax, Radiography, Thoracic, Language. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
This study evaluates the diagnostic performance of three multimodal large language models (LLMs)—ChatGPT-4o, Gemini 2.0, and Claude 3.5—in identifying pneumothorax from chest radiographs.
Methods
In this retrospective analysis, 172 pneumothorax cases (148 patients aged >12 years, 24 patients aged ≤12 years) with both chest radiographs and confirmatory thoracic CT were included from a tertiary emergency department. Patients were categorized by age and pneumothorax size (small/large). Each radiograph was presented to all three LLMs accompanied by basic symptoms (dyspnea or chest pain), with each model analyzing each image three times. Diagnostic accuracy was evaluated using overall accuracy (all three responses correct), strict accuracy (≥2 responses correct), and ideal accuracy (≥1 response correct), alongside response consistency assessment using Fleiss’ Kappa.
Results
In patients older than 12 years, ChatGPT-4o demonstrated the highest overall accuracy (69.6%), followed by Claude 3.5 (64.9%) and Gemini 2.0 (57.4%). Performance was significantly poorer in pediatric patients across all models (20.8%, 12.5%, and 20.8%, respectively). For large pneumothorax in adults, ChatGPT-4o showed significantly higher accuracy compared to small pneumothorax (81.6% vs. 42.2%; p < 0.001). Regarding consistency, Gemini 2.0 demonstrated excellent reliability for large pneumothorax (Kappa = 1.00), while Claude 3.5 showed moderate consistency across both pneumothorax sizes.
Conclusion
This study, the first to evaluate these three current multimodal LLMs in pneumothorax identification across different age groups, demonstrates promising results for potential clinical applications, particularly for adult patients with large pneumothorax. However, performance limitations in pediatric cases and with small pneumothoraces highlight the need for further validation before clinical implementation.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Emergency Medicine, Ankara Gulhane Training and Research Hospital, Health Science University, Ankara, Turkey.
- Department of Emergency Medicine, Ankara Training and Research Hospital, Ankara, Turkey.
- Department of Emergency Medicine, Etimesgut Şehit Sait Ertürk State Hospital, Ankara, Turkey.
- Department Of Emergency Medicine, Konya Numune Hospital, Konya, Turkey.
- Department of Internal Medicine, Konya City Hospital, University of Health Sciences, Konya, Turkey.
- Department of Anesthesıology And Reanımatıon Department, 75th Year Oral And Dental Health Hospital, Ankara, Turkey.
ORCID-Profile: Bensu Bulut, Ayşenur Gür, Hüseyin Mutlu
Zitieren
APA
Bulut, B., Öz, M. A., Genç, M., Gür, A., Yortanlı, M., Yortanlı, B. Ç., Sariyildiz, O., Yazıcı, R., Mutlu, H., Kotanoglu, M. S., Cinar, E., & Uykan, Z. (2025). New frontiers in radiologic interpretation: evaluating the effectiveness of large language models in pneumothorax diagnosis. PLOS One. https://doi.org/10.1371/journal.pone.0331962
Vancouver
Bulut B, Öz MA, Genç M, Gür A, Yortanlı M, Yortanlı BÇ, et al. New frontiers in radiologic interpretation: evaluating the effectiveness of large language models in pneumothorax diagnosis. PLOS One. 2025. doi:10.1371/journal.pone.0331962
BibTeX
@article{Bulut2025,
author = {Bulut, Bensu and Öz, Medine Akkan and Genç, Murat and Gür, Ayşenur and Yortanlı, Mehmet and Yortanlı, Betül Çiğdem and Sariyildiz, Oguz and Yazıcı, Ramiz and Mutlu, Hüseyin and Kotanoglu, Mustafa Sirri and Cinar, Eray and Uykan, Zekeriya},
title = {New frontiers in radiologic interpretation: evaluating the effectiveness of large language models in pneumothorax diagnosis},
journal = {PLOS One},
year = {2025},
doi = {10.1371/journal.pone.0331962}
}
EUVIMED