Evaluating prompt and data perturbation sensitivity in large language models for radiology reports classification
JAMIA Open · 03.07.2025 · 8 (4)
DOI: 10.1093/jamiaopen/ooaf073 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 12.08.2025. Verfasst von Vera Sorin und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, Mayo Clinic College of Medicine and Science, Mayo Clinic, Rochester, MN 55905, United States. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Objectives
Large language models (LLMs) offer potential in natural language processing tasks in healthcare. Due to the need for high accuracy, understanding their limitations is essential. The purpose of this study was to evaluate the performance of LLMs in classifying radiology reports for the presence of pulmonary embolism (PE) under various conditions, including different prompt designs and data perturbations. Materials and Methods In this retrospective, institutional review board approved study, we evaluated 3 Google’s LLMs including Gemini-1.5-Pro, Gemini-1.5-Flash-001, and Gemini-1.5-Flash-002, in classifying 11 999 pulmonary CT angiography radiology reports for PE. Ground truth labels were determined by concordance between a computer vision-based PE detection (CVPED) algorithm and multiple LLM runs under various configurations. Discrepancies between algorithms’ classifications were aggregated and manually reviewed. We evaluated the effects of prompt design, data perturbations, and repeated analyses across geographic cloud regions. Performance metrics were calculated.
Results
Of 11 999 reports, 1296 (10.8%) were PE-positive. Accuracy across LLMs ranged between 0.953 and 0.996. The highest recall rate for a prompt modified after a review of the misclassified cases (up to 0.997). Few-shot prompting improved recall (up to 0.99), while chain-of-thought generally degraded performance. Gemini-1.5-Flash-002 demonstrated the highest robustness against data perturbations. Geographic cloud region variability was minimal for Gemini-1.5+-Pro, while the Flash models showed stable performance. Discussion and Conclusion LLMs demonstrated high performance in classifying radiology reports, though results varied with prompt design and data quality. These findings underscore the need for systematic evaluation and validation of LLMs for clinical applications, particularly in high-stakes scenarios.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Radiology, Mayo Clinic College of Medicine and Science, Mayo Clinic, Rochester, MN 55905, United States.
ORCID-Profile: Vera Sorin, Jeremy D Collins
Zitieren
APA
Sorin, V., Collins, J. D., Bratt, A. K., Kusmirek, J. E., Mugu, V. K., Kline, T. L., Butler, C. L., Wood, N. G., Cook, C. J., & Korfiatis, P. (2025). Evaluating prompt and data perturbation sensitivity in large language models for radiology reports classification. JAMIA Open, 8(4). https://doi.org/10.1093/jamiaopen/ooaf073
Vancouver
Sorin V, Collins JD, Bratt AK, Kusmirek JE, Mugu VK, Kline TL, et al. Evaluating prompt and data perturbation sensitivity in large language models for radiology reports classification. JAMIA Open. 2025;8(4). doi:10.1093/jamiaopen/ooaf073
BibTeX
@article{Sorin2025,
author = {Sorin, Vera and Collins, Jeremy D and Bratt, Alex K and Kusmirek, Joanna E and Mugu, Vamshi K and Kline, Timothy L and Butler, Crystal L and Wood, Nadia G and Cook, Cole J and Korfiatis, Panagiotis},
title = {Evaluating prompt and data perturbation sensitivity in large language models for radiology reports classification},
journal = {JAMIA Open},
year = {2025},
volume = {8},
number = {4},
doi = {10.1093/jamiaopen/ooaf073}
}
EUVIMED