EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

New frontiers in radiologic interpretation: evaluating the effectiveness of large language models in pneumothorax diagnosis

Bensu Bulut iD, Medine Akkan Öz, Murat Genç, Ayşenur Gür iD, Mehmet Yortanlı, Betül Çiğdem Yortanlı, Oguz Sariyildiz, Ramiz Yazıcı, Hüseyin Mutlu iD, Mustafa Sirri Kotanoglu, Eray Cinar, Zekeriya Uykan

PLOS One · 2025

DOI: 10.1371/journal.pone.0331962 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, 2025. Verfasst von Bensu Bulut und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Emergency Medicine, Ankara Gulhane Training and Research Hospital, Health Science University, Ankara, Turkey. Hauptthemen laut MEDLINE: Pneumothorax, Radiography, Thoracic, Language. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

This study evaluates the diagnostic performance of three multimodal large language models (LLMs)—ChatGPT-4o, Gemini 2.0, and Claude 3.5—in identifying pneumothorax from chest radiographs.

Methods

In this retrospective analysis, 172 pneumothorax cases (148 patients aged >12 years, 24 patients aged ≤12 years) with both chest radiographs and confirmatory thoracic CT were included from a tertiary emergency department. Patients were categorized by age and pneumothorax size (small/large). Each radiograph was presented to all three LLMs accompanied by basic symptoms (dyspnea or chest pain), with each model analyzing each image three times. Diagnostic accuracy was evaluated using overall accuracy (all three responses correct), strict accuracy (≥2 responses correct), and ideal accuracy (≥1 response correct), alongside response consistency assessment using Fleiss’ Kappa.

Results

In patients older than 12 years, ChatGPT-4o demonstrated the highest overall accuracy (69.6%), followed by Claude 3.5 (64.9%) and Gemini 2.0 (57.4%). Performance was significantly poorer in pediatric patients across all models (20.8%, 12.5%, and 20.8%, respectively). For large pneumothorax in adults, ChatGPT-4o showed significantly higher accuracy compared to small pneumothorax (81.6% vs. 42.2%; p < 0.001). Regarding consistency, Gemini 2.0 demonstrated excellent reliability for large pneumothorax (Kappa = 1.00), while Claude 3.5 showed moderate consistency across both pneumothorax sizes.

Conclusion

This study, the first to evaluate these three current multimodal LLMs in pneumothorax identification across different age groups, demonstrates promising results for potential clinical applications, particularly for adult patients with large pneumothorax. However, performance limitations in pediatric cases and with small pneumothoraces highlight the need for further validation before clinical implementation.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

PneumothoraxRadiography, ThoracicLanguageHumansMaleFemaleChildRetrospective StudiesAdolescentAdultYoung AdultTomography, X-Ray ComputedMiddle AgedChild, PreschoolLarge Language Models

Autor:innen und Institutionen

  • Department of Emergency Medicine, Ankara Gulhane Training and Research Hospital, Health Science University, Ankara, Turkey.
  • Department of Emergency Medicine, Ankara Training and Research Hospital, Ankara, Turkey.
  • Department of Emergency Medicine, Etimesgut Şehit Sait Ertürk State Hospital, Ankara, Turkey.
  • Department Of Emergency Medicine, Konya Numune Hospital, Konya, Turkey.
  • Department of Internal Medicine, Konya City Hospital, University of Health Sciences, Konya, Turkey.
  • Department of Anesthesıology And Reanımatıon Department, 75th Year Oral And Dental Health Hospital, Ankara, Turkey.

ORCID-Profile: Bensu Bulut, Ayşenur Gür, Hüseyin Mutlu

Zitieren

APA

Bulut, B., Öz, M. A., Genç, M., Gür, A., Yortanlı, M., Yortanlı, B. Ç., Sariyildiz, O., Yazıcı, R., Mutlu, H., Kotanoglu, M. S., Cinar, E., & Uykan, Z. (2025). New frontiers in radiologic interpretation: evaluating the effectiveness of large language models in pneumothorax diagnosis. PLOS One. https://doi.org/10.1371/journal.pone.0331962

Vancouver

Bulut B, Öz MA, Genç M, Gür A, Yortanlı M, Yortanlı BÇ, et al. New frontiers in radiologic interpretation: evaluating the effectiveness of large language models in pneumothorax diagnosis. PLOS One. 2025. doi:10.1371/journal.pone.0331962

BibTeX
@article{Bulut2025,
  author = {Bulut, Bensu and Öz, Medine Akkan and Genç, Murat and Gür, Ayşenur and Yortanlı, Mehmet and Yortanlı, Betül Çiğdem and Sariyildiz, Oguz and Yazıcı, Ramiz and Mutlu, Hüseyin and Kotanoglu, Mustafa Sirri and Cinar, Eray and Uykan, Zekeriya},
  title = {New frontiers in radiologic interpretation: evaluating the effectiveness of large language models in pneumothorax diagnosis},
  journal = {PLOS One},
  year = {2025},
  doi = {10.1371/journal.pone.0331962}
}
Kostenlos starten →