A comparative performance analysis of regular expressions and a large language model-based approach to extract the BI-RADS score from radiological reports
JAMIA Open · 03.11.2025 · 8 (6)
DOI: 10.1093/jamiaopen/ooaf128 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 12.11.2025. Verfasst von Fabio Dennstädt und 12 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern 3010, Switzerland. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Innosuisse.
Abstract (Originalsprache)
Background
Different natural language processing (NLP) techniques have demonstrated promising results for data extraction from radiological reports. Both traditional rule-based methods like regular expressions (Regex) and modern large language models (LLMs) can extract structured information. However, comparison between these approaches for extraction of specific radiological data elements has not been widely conducted.
Methods
We compared accuracy and processing time between Regex and LLM-based approaches for extracting Breast Imaging-Reporting and Data System (BI-RADS) scores from 7764 radiology reports (mammography, ultrasound, MRI [magnetic resonance imaging], and biopsy). We developed a rule-based algorithm using Regex patterns and implemented an LLM-based extraction using the Rombos-LLM-V2.6-Qwen-14b model. A ground truth dataset of 199 manually classified reports was used for evaluation.
Results
We did not detect a statistically significant difference in the accuracy in extracting BI-RADS scores between Regex and an LLM-based method (accuracy of 89.20% for Regex vs 87.69% for the LLM-based method; P = .56, effect size w = 0.04; post-hoc power = 0.11). Compared to the LLM-based method, Regex processing was more efficient, completing the task 28 120 times faster (0.06 seconds vs 1687.20 seconds). Further analysis revealed that LLMs favored common classifications (particularly BI-RADS value of 2) while Regex more frequently returned “unclear” values. We also could confirm in our sample an already known laterality bias for breast cancer (BI-RADS 6) and detected a slight laterality skew for suspected breast cancer (BI-RADS 5) as well.
Conclusion
For structured, standardized data like BI-RADS, traditional NLP techniques seem to be superior, though future work should explore hybrid approaches combining Regex precision for standardized elements with LLM contextual understanding for more complex information extraction tasks.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern 3010, Switzerland.
- School of Medicine, University of St. Gallen, St. Gallen 9000, Switzerland.
- Department of Diagnostic, Interventional and Pediatric Radiology (DIPR), Inselspital, Bern University Hospital, University of Bern, Bern 3010, Switzerland.
- Department of Radiology, Lindenhof Hospital, Bern 3012, Switzerland.
- Department of Neurology, Inselspital, Bern University Hospital and University of Bern, Bern 3010, Switzerland.
- Institute for Implementation Science in Health Care, University of Zurich, Zürich 8006, Switzerland.
ORCID-Profile: Fabio Dennstädt, Janna Hastings, Knud Nairz
Förderung
- Innosuisse
Zitieren
APA
Dennstädt, F., Lerch, L., Schmerder, M., Cihoric, N., Cereghetti, G. M., Gaio, R., Bonel, H., Filchenko, I., Hastings, J., Dammann, F., Aebersold, D. M., von Tengg-Kobligk, H., & Nairz, K. (2025). A comparative performance analysis of regular expressions and a large language model-based approach to extract the BI-RADS score from radiological reports. JAMIA Open, 8(6). https://doi.org/10.1093/jamiaopen/ooaf128
Vancouver
Dennstädt F, Lerch L, Schmerder M, Cihoric N, Cereghetti GM, Gaio R, et al. A comparative performance analysis of regular expressions and a large language model-based approach to extract the BI-RADS score from radiological reports. JAMIA Open. 2025;8(6). doi:10.1093/jamiaopen/ooaf128
BibTeX
@article{Dennstdt2025,
author = {Dennstädt, Fabio and Lerch, Luc and Schmerder, Max and Cihoric, Nikola and Cereghetti, Grazia Maria and Gaio, Roberto and Bonel, Harald and Filchenko, Irina and Hastings, Janna and Dammann, Florian and Aebersold, Daniel M and von Tengg-Kobligk, Hendrik and Nairz, Knud},
title = {A comparative performance analysis of regular expressions and a large language model-based approach to extract the BI-RADS score from radiological reports},
journal = {JAMIA Open},
year = {2025},
volume = {8},
number = {6},
doi = {10.1093/jamiaopen/ooaf128}
}
EUVIMED