EUVIMEDEUVIMED entdecken
Systematische Übersicht

Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening

Siddhant Dogra, Soroush Arabshahi, Jason Wei, Emmy Hu, Lucia Saidenberg, Sonali Sharma, Zehui Gu, Mutita Siriruchatanon, Stella K Kang

Academic Radiology · 2025

DOI: 10.1016/j.acra.2025.08.014 ↗

Kurz gefasst

Systematische Übersicht in der Zeitschrift Academic Radiology, 2025. Verfasst von Siddhant Dogra und 8 weiteren Autor:innen. Beteiligte Einrichtung: New York University Langone Health. Department of Radiology. New York, NY (S.D., J.W., E.H.). Electronic address: Siddhant.Dogra@nyulangone.org. Hauptthemen laut MEDLINE: Systematic Reviews as Topic, Radiology, Natural Language Processing, Language. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 27 Literaturangaben. Gefördert von: U.S. Department of Defense, American Roentgen Ray Society, National Institutes of Health.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

RATIONALE AND OBJECTIVES: To evaluate the performance, stability, and decision-making behavior of large language models (LLMs) for title and abstract screening for radiology systematic reviews, with attention to prompt framing, confidence calibration, and model robustness under disagreement. MATERIALS AND METHODS: We compared five LLMs (GPT-4o, GPT-4o mini, Gemini 1.5 Pro, Gemini 2.0 Flash, Llama 3.3 70B) on two imaging-focused systematic reviews (n = 5438 and n = 267 abstracts) using binary and ternary classification tasks, confidence scoring, and reclassification of true and synthetic disagreements. Disagreements were framed as either "LLM vs human" or "human vs human." We also piloted autonomous PubMed retrieval using OpenAI and Gemini Deep Research tools. RESULTS: LLMs achieved high specificity and variable sensitivity across reviews and tasks, with F1 scores ranging from 0.389 to 0.854. Ternary classification showed low abstention rates (

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Systematic Reviews as TopicRadiologyNatural Language ProcessingLanguageHumansSensitivity and SpecificityAbstracting and IndexingLarge Language Models

Stichwörter der Autor:innen

Large language modelsSystematic reviewsTitle and abstract screening

Autor:innen und Institutionen

  • New York University Langone Health. Department of Radiology. New York, NY (S.D., J.W., E.H.). Electronic address: Siddhant.Dogra@nyulangone.org.
  • Columbia University, Department of Biomedical Engineering. New York, NY (S.A.).
  • New York University Langone Health. Department of Radiology. New York, NY (S.D., J.W., E.H.).
  • New York University Langone Health. Department of Neurology. New York, NY (L.S.).
  • Department of Radiology, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada (S.S.).
  • Columbia University, Department of Radiology. New York, NY (Z.G., M.S., S.K.K.).

Förderung

  • U.S. Department of Defense
  • American Roentgen Ray Society
  • National Institutes of Health
  • National Institute of Biomedical Imaging and Bioengineering

Zitieren

APA

Dogra, S., Arabshahi, S., Wei, J., Hu, E., Saidenberg, L., Sharma, S., Gu, Z., Siriruchatanon, M., & Kang, S. K. (2025). Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening. Academic Radiology. https://doi.org/10.1016/j.acra.2025.08.014

Vancouver

Dogra S, Arabshahi S, Wei J, Hu E, Saidenberg L, Sharma S, et al. Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening. Academic Radiology. 2025. doi:10.1016/j.acra.2025.08.014

BibTeX
@article{Dogra2025,
  author = {Dogra, Siddhant and Arabshahi, Soroush and Wei, Jason and Hu, Emmy and Saidenberg, Lucia and Sharma, Sonali and Gu, Zehui and Siriruchatanon, Mutita and Kang, Stella K},
  title = {Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening},
  journal = {Academic Radiology},
  year = {2025},
  doi = {10.1016/j.acra.2025.08.014}
}
Kostenlos starten →