EUVIMEDEUVIMED entdecken
Multizentrische Studie

Applying large language model for automated quality scoring of radiology requisitions using a standardized criteria

Raşit Eren Büyüktoka, Murat Surucu, Pelin Berfin Erekli Derinkaya, Zehra Hilal Adibelli, Ali Salbas, Ali Murat Koc, Asli Dilara Buyuktoka, Yalcın Isler, Mehmet Alperen Ugur, Elif Isiklar

European Radiology · 2025

DOI: 10.1007/s00330-025-11933-2 ↗

Kurz gefasst

Multizentrische Studie in der Zeitschrift European Radiology, 2025. Verfasst von Raşit Eren Büyüktoka und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, Izmir Foça State Hospital, Izmir, Türkiye. rasiterenbuyuktoka@hotmail.com. Hauptthemen laut MEDLINE: Radiology Information Systems, Radiology. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 34 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

OBJECTIVES: To create and test a locally adapted large language model (LLM) for automated scoring of radiology requisitions based on the reason for exam imaging reporting and data system (RI-RADS), and to evaluate its performance based on reference standards. MATERIALS AND METHODS: This retrospective, double-center study included 131,683 radiology requisitions from two institutions. A bidirectional encoder representation from a transformer (BERT)-based model was trained using 101,563 requisitions from Center 1 (including 1500 synthetic examples) and externally tested on 18,887 requisitions from Center 2. The model's performance for two different classification strategies was evaluated by the reference standard created by three different radiologists. Model performance was assessed using Cohen's Kappa, accuracy, F1-score, sensitivity, and specificity with 95% confidence intervals. RESULTS: A total of 18,887 requisitions were evaluated for the external test set. External testing yielded a performance with an F1-score of 0.93 (95% CI: 0.912-0.943); κ = 0.88 (95% CI: 0.871-0.884). Performance was highest in common categories RI-RADS D and X (F1 ≥ 0.96) and lowest for rare categories RI-RADS A and B (F1 ≤ 0.49). When grouped into three categories (adequate, inadequate, and unacceptable), overall model performance improved [F1-score = 0.97; (95% CI: 0.96-0.97)]. CONCLUSION: The locally adapted BERT-based model demonstrated high performance and almost perfect agreement with radiologists in automated RI-RADS scoring, showing promise for integration into radiology workflows to improve requisition completeness and communication. KEY POINTS: Question Can an LLM accurately and automatically score radiology requisitions based on standardized criteria to address the challenges of incomplete information in radiological practice? Findings A locally adapted BERT-based model demonstrated high performance (F1-score 0.93) and almost perfect agreement with radiologists in automated RI-RADS scoring across a large, multi-institutional dataset. Clinical relevance LLMs offer a scalable solution for automated scoring of radiology requisitions, with the potential to improve workflow in radiology. Further improvement and integration into clinical practice could enhance communication, contributing to better diagnoses and patient care.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Radiology Information SystemsRadiologyHumansRetrospective StudiesSensitivity and SpecificityReference StandardsLarge Language Models

Stichwörter der Autor:innen

Electronic health recordsLarge language modelsNatural language processingRadiologyWorkflow

Autor:innen und Institutionen

  • Department of Radiology, Izmir Foça State Hospital, Izmir, Türkiye. rasiterenbuyuktoka@hotmail.com.
  • Bucak Computer and Informatics Faculty, Burdur Mehmet Akif Ersoy University, Burdur, Türkiye.
  • Department of Radiology, Izmir City Hospital, Izmir, Türkiye.
  • Department of Radiology, Izmir City Hospital & Health Sciences University, Izmir, Türkiye.
  • Department of Radiology, Izmir Ataturk Education and Research Hospital, Izmir, Türkiye.
  • Department of Radiology, Izmir Katip Celebi University & Izmir Ataturk Education and Research Hospital, Izmir, Türkiye.

Zitieren

APA

Büyüktoka, R. E., Surucu, M., Erekli Derinkaya, P. B., Adibelli, Z. H., Salbas, A., Koc, A. M., Buyuktoka, A. D., Isler, Y., Ugur, M. A., & Isiklar, E. (2025). Applying large language model for automated quality scoring of radiology requisitions using a standardized criteria. European Radiology. https://doi.org/10.1007/s00330-025-11933-2

Vancouver

Büyüktoka RE, Surucu M, Erekli Derinkaya PB, Adibelli ZH, Salbas A, Koc AM, et al. Applying large language model for automated quality scoring of radiology requisitions using a standardized criteria. European Radiology. 2025. doi:10.1007/s00330-025-11933-2

BibTeX
@article{Byktoka2025,
  author = {Büyüktoka, Raşit Eren and Surucu, Murat and Erekli Derinkaya, Pelin Berfin and Adibelli, Zehra Hilal and Salbas, Ali and Koc, Ali Murat and Buyuktoka, Asli Dilara and Isler, Yalcın and Ugur, Mehmet Alperen and Isiklar, Elif},
  title = {Applying large language model for automated quality scoring of radiology requisitions using a standardized criteria},
  journal = {European Radiology},
  year = {2025},
  doi = {10.1007/s00330-025-11933-2}
}
Kostenlos starten →