Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening
Academic Radiology · 2025
DOI: 10.1016/j.acra.2025.08.014 ↗Kurz gefasst
Systematische Übersicht in der Zeitschrift Academic Radiology, 2025. Verfasst von Siddhant Dogra und 8 weiteren Autor:innen. Beteiligte Einrichtung: New York University Langone Health. Department of Radiology. New York, NY (S.D., J.W., E.H.). Electronic address: Siddhant.Dogra@nyulangone.org. Hauptthemen laut MEDLINE: Systematic Reviews as Topic, Radiology, Natural Language Processing, Language. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 27 Literaturangaben. Gefördert von: U.S. Department of Defense, American Roentgen Ray Society, National Institutes of Health.
Abstract (Originalsprache)
RATIONALE AND OBJECTIVES: To evaluate the performance, stability, and decision-making behavior of large language models (LLMs) for title and abstract screening for radiology systematic reviews, with attention to prompt framing, confidence calibration, and model robustness under disagreement. MATERIALS AND METHODS: We compared five LLMs (GPT-4o, GPT-4o mini, Gemini 1.5 Pro, Gemini 2.0 Flash, Llama 3.3 70B) on two imaging-focused systematic reviews (n = 5438 and n = 267 abstracts) using binary and ternary classification tasks, confidence scoring, and reclassification of true and synthetic disagreements. Disagreements were framed as either "LLM vs human" or "human vs human." We also piloted autonomous PubMed retrieval using OpenAI and Gemini Deep Research tools. RESULTS: LLMs achieved high specificity and variable sensitivity across reviews and tasks, with F1 scores ranging from 0.389 to 0.854. Ternary classification showed low abstention rates (
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- New York University Langone Health. Department of Radiology. New York, NY (S.D., J.W., E.H.). Electronic address: Siddhant.Dogra@nyulangone.org.
- Columbia University, Department of Biomedical Engineering. New York, NY (S.A.).
- New York University Langone Health. Department of Radiology. New York, NY (S.D., J.W., E.H.).
- New York University Langone Health. Department of Neurology. New York, NY (L.S.).
- Department of Radiology, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada (S.S.).
- Columbia University, Department of Radiology. New York, NY (Z.G., M.S., S.K.K.).
Förderung
- U.S. Department of Defense
- American Roentgen Ray Society
- National Institutes of Health
- National Institute of Biomedical Imaging and Bioengineering
Zitieren
APA
Dogra, S., Arabshahi, S., Wei, J., Hu, E., Saidenberg, L., Sharma, S., Gu, Z., Siriruchatanon, M., & Kang, S. K. (2025). Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening. Academic Radiology. https://doi.org/10.1016/j.acra.2025.08.014
Vancouver
Dogra S, Arabshahi S, Wei J, Hu E, Saidenberg L, Sharma S, et al. Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening. Academic Radiology. 2025. doi:10.1016/j.acra.2025.08.014
BibTeX
@article{Dogra2025,
author = {Dogra, Siddhant and Arabshahi, Soroush and Wei, Jason and Hu, Emmy and Saidenberg, Lucia and Sharma, Sonali and Gu, Zehui and Siriruchatanon, Mutita and Kang, Stella K},
title = {Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening},
journal = {Academic Radiology},
year = {2025},
doi = {10.1016/j.acra.2025.08.014}
}
EUVIMED