Large Language Model Automated Extraction of Clinical Signs and Symptoms From Emergency Department Reports for Machine Learning Prediction Models: Development and Validation Study
JMIR Medical Informatics · 2026
DOI: 10.2196/81500 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift JMIR Medical Informatics, 2026. Verfasst von Anoeska Schipper und 9 weiteren Autor:innen. Beteiligte Einrichtung: Diagnostic Image Analysis Group, Medical Imaging Department, Radboud University Medical Center, Geert Grooteplein Zuid 10, Nijmegen, 6525 GA, The Net…. Hauptthemen laut MEDLINE: Large Language Models, Emergency Service, Hospital, Machine Learning. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Most clinically relevant information in emergency department (ED) visits is documented in free text, limiting reuse for research and clinical decision support. Despite growing interest in large language model (LLM)–based feature extraction, very few studies have examined it directly on ED reports. Existing work has mainly addressed binary tasks and rarely evaluated their impact on downstream prediction models. Furthermore, evidence for small multilingual LLMs remains limited, especially for underrepresented languages such as Dutch. Locally deployable LLMs could enable automated feature extraction for decision support systems without increasing physician workload.
Objective
We aim to evaluate whether a small open-source LLM (Qwen 2.5:14B) can automatically extract 16 clinical signs and symptoms from ED reports and use these as input for an appendicitis prediction model. LLM performance under minimal and optimized 0-shot prompts was assessed against researcher annotations (reference standard) and physician annotations.
Methods
This retrospective study used 336 ED reports from patients presenting with acute abdominal pain to a Dutch teaching hospital (2016-2023). One hundred reports were randomly selected to develop a minimal and an optimized 0-shot prompt strategy. The remaining 236 reports, reserved for evaluation, were annotated by 2 ED physicians and processed by the LLM to extract 16 signs and symptoms, covering binary, multiclass, and multilabel classification tasks. These features were used as input to the HIVE (History, Intake, Vitals, Examination) appendicitis prediction model. LLM extraction accuracy, sensitivity, and specificity were measured against the researcher’s (reference standard) and physician annotations. The HIVE model’s area under the receiver operating characteristic curve was evaluated using LLM-extracted vs physician-annotated features.
Results
Among 336 ED reports from patients with acute abdominal pain (median age 41, IQR 22‐62 years, 205/336, 61% female), 50% (167/336) had appendicitis. The LLM achieved weighted average accuracies of 0.910 (95% CI (0.018) with minimal prompts and 0.929 (95% CI ±0.016) with optimized prompts, vs 0.961 (95% CI ±0.012) and 0.951 (95% CI ±0.015) for physicians. Corresponding HIVE model area under the receiver operating characteristic curves were 0.871 (95% CI ±0.019) and 0.911 (95% CI ±0.014) with LLM inputs under the minimal and optimized prompts, compared to 0.917 (95% CI ±0.015) and 0.924 (95% CI ±0.018) for physician inputs.
Conclusions
A small locally deployable multilingual LLM can approach physician-level accuracy in extracting structured binary, multiclass, and multilabel clinical data from free-text Dutch ED reports, while preserving patient privacy, interpretability, and statistical transparency for downstream diagnostic modeling.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Diagnostic Image Analysis Group, Medical Imaging Department, Radboud University Medical Center, Geert Grooteplein Zuid 10, Nijmegen, 6525 GA, The Netherlands, 31 614021323.
- Laboratory of Clinical Chemistry and Hematology, Jeroen Bosch Ziekenhuis, Den Bosch, The Netherlands.
- Psychiatry Department, Radboud University Medical Center, Nijmegen, The Netherlands.
- Emergency Department, Jeroen Bosch Ziekenhuis, Den Bosch, The Netherlands.
- Department of Health Technology and Services Research, Technical Medical Centre, University of Twente, Enschede, The Netherlands.
- Laboratory of Clinical Chemistry and Laboratory Medicine, Dicoon BV, Canisius-Wilhelmina Ziekenhuis, Nijmegen, The Netherlands.
ORCID-Profile: Anoeska Schipper, Peter Belgers, Rory David O'Connor, Lieke van de Wouw, Luc Builtjes, Joeran S Bosma, Ron Kusters, Steef Kurstjens, Matthieu Rutten, Bram van Ginneken
Zitieren
APA
Schipper, A., Belgers, P., O'Connor, R. D., van de Wouw, L., Builtjes, L., Bosma, J. S., Kusters, R., Kurstjens, S., Rutten, M., & van Ginneken, B. (2026). Large Language Model Automated Extraction of Clinical Signs and Symptoms From Emergency Department Reports for Machine Learning Prediction Models: Development and Validation Study. JMIR Medical Informatics. https://doi.org/10.2196/81500
Vancouver
Schipper A, Belgers P, O'Connor RD, van de Wouw L, Builtjes L, Bosma JS, et al. Large Language Model Automated Extraction of Clinical Signs and Symptoms From Emergency Department Reports for Machine Learning Prediction Models: Development and Validation Study. JMIR Medical Informatics. 2026. doi:10.2196/81500
BibTeX
@article{Schipper2026,
author = {Schipper, Anoeska and Belgers, Peter and O'Connor, Rory David and van de Wouw, Lieke and Builtjes, Luc and Bosma, Joeran S and Kusters, Ron and Kurstjens, Steef and Rutten, Matthieu and van Ginneken, Bram},
title = {Large Language Model Automated Extraction of Clinical Signs and Symptoms From Emergency Department Reports for Machine Learning Prediction Models: Development and Validation Study},
journal = {JMIR Medical Informatics},
year = {2026},
doi = {10.2196/81500}
}
EUVIMED