EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Evaluating prompt and data perturbation sensitivity in large language models for radiology reports classification

Vera Sorin iD, Jeremy D Collins iD, Alex K Bratt, Joanna E Kusmirek, Vamshi K Mugu, Timothy L Kline, Crystal L Butler, Nadia G Wood, Cole J Cook, Panagiotis Korfiatis

JAMIA Open · 03.07.2025 · 8 (4)

DOI: 10.1093/jamiaopen/ooaf073 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 12.08.2025. Verfasst von Vera Sorin und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, Mayo Clinic College of Medicine and Science, Mayo Clinic, Rochester, MN 55905, United States. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Large language models (LLMs) offer potential in natural language processing tasks in healthcare. Due to the need for high accuracy, understanding their limitations is essential. The purpose of this study was to evaluate the performance of LLMs in classifying radiology reports for the presence of pulmonary embolism (PE) under various conditions, including different prompt designs and data perturbations. Materials and Methods In this retrospective, institutional review board approved study, we evaluated 3 Google’s LLMs including Gemini-1.5-Pro, Gemini-1.5-Flash-001, and Gemini-1.5-Flash-002, in classifying 11 999 pulmonary CT angiography radiology reports for PE. Ground truth labels were determined by concordance between a computer vision-based PE detection (CVPED) algorithm and multiple LLM runs under various configurations. Discrepancies between algorithms’ classifications were aggregated and manually reviewed. We evaluated the effects of prompt design, data perturbations, and repeated analyses across geographic cloud regions. Performance metrics were calculated.

Results

Of 11 999 reports, 1296 (10.8%) were PE-positive. Accuracy across LLMs ranged between 0.953 and 0.996. The highest recall rate for a prompt modified after a review of the misclassified cases (up to 0.997). Few-shot prompting improved recall (up to 0.99), while chain-of-thought generally degraded performance. Gemini-1.5-Flash-002 demonstrated the highest robustness against data perturbations. Geographic cloud region variability was minimal for Gemini-1.5+-Pro, while the Flash models showed stable performance. Discussion and Conclusion LLMs demonstrated high performance in classifying radiology reports, though results varied with prompt design and data quality. These findings underscore the need for systematic evaluation and validation of LLMs for clinical applications, particularly in high-stakes scenarios.

Schlagwörter

Stichwörter der Autor:innen

artificial intelligenceclassificationlarge language modelpulmonary embolism

Autor:innen und Institutionen

  • Department of Radiology, Mayo Clinic College of Medicine and Science, Mayo Clinic, Rochester, MN 55905, United States.

ORCID-Profile: Vera Sorin, Jeremy D Collins

Zitieren

APA

Sorin, V., Collins, J. D., Bratt, A. K., Kusmirek, J. E., Mugu, V. K., Kline, T. L., Butler, C. L., Wood, N. G., Cook, C. J., & Korfiatis, P. (2025). Evaluating prompt and data perturbation sensitivity in large language models for radiology reports classification. JAMIA Open, 8(4). https://doi.org/10.1093/jamiaopen/ooaf073

Vancouver

Sorin V, Collins JD, Bratt AK, Kusmirek JE, Mugu VK, Kline TL, et al. Evaluating prompt and data perturbation sensitivity in large language models for radiology reports classification. JAMIA Open. 2025;8(4). doi:10.1093/jamiaopen/ooaf073

BibTeX
@article{Sorin2025,
  author = {Sorin, Vera and Collins, Jeremy D and Bratt, Alex K and Kusmirek, Joanna E and Mugu, Vamshi K and Kline, Timothy L and Butler, Crystal L and Wood, Nadia G and Cook, Cole J and Korfiatis, Panagiotis},
  title = {Evaluating prompt and data perturbation sensitivity in large language models for radiology reports classification},
  journal = {JAMIA Open},
  year = {2025},
  volume = {8},
  number = {4},
  doi = {10.1093/jamiaopen/ooaf073}
}
Kostenlos starten →