EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

Large Language Model Automated Extraction of Clinical Signs and Symptoms From Emergency Department Reports for Machine Learning Prediction Models: Development and Validation Study

Anoeska Schipper iD, Peter Belgers iD, Rory David O'Connor iD, Lieke van de Wouw iD, Luc Builtjes iD, Joeran S Bosma iD, Ron Kusters iD, Steef Kurstjens iD, Matthieu Rutten iD, Bram van Ginneken iD

JMIR Medical Informatics · 2026

DOI: 10.2196/81500 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift JMIR Medical Informatics, 2026. Verfasst von Anoeska Schipper und 9 weiteren Autor:innen. Beteiligte Einrichtung: Diagnostic Image Analysis Group, Medical Imaging Department, Radboud University Medical Center, Geert Grooteplein Zuid 10, Nijmegen, 6525 GA, The Net…. Hauptthemen laut MEDLINE: Large Language Models, Emergency Service, Hospital, Machine Learning. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Most clinically relevant information in emergency department (ED) visits is documented in free text, limiting reuse for research and clinical decision support. Despite growing interest in large language model (LLM)–based feature extraction, very few studies have examined it directly on ED reports. Existing work has mainly addressed binary tasks and rarely evaluated their impact on downstream prediction models. Furthermore, evidence for small multilingual LLMs remains limited, especially for underrepresented languages such as Dutch. Locally deployable LLMs could enable automated feature extraction for decision support systems without increasing physician workload.

Objective

We aim to evaluate whether a small open-source LLM (Qwen 2.5:14B) can automatically extract 16 clinical signs and symptoms from ED reports and use these as input for an appendicitis prediction model. LLM performance under minimal and optimized 0-shot prompts was assessed against researcher annotations (reference standard) and physician annotations.

Methods

This retrospective study used 336 ED reports from patients presenting with acute abdominal pain to a Dutch teaching hospital (2016-2023). One hundred reports were randomly selected to develop a minimal and an optimized 0-shot prompt strategy. The remaining 236 reports, reserved for evaluation, were annotated by 2 ED physicians and processed by the LLM to extract 16 signs and symptoms, covering binary, multiclass, and multilabel classification tasks. These features were used as input to the HIVE (History, Intake, Vitals, Examination) appendicitis prediction model. LLM extraction accuracy, sensitivity, and specificity were measured against the researcher’s (reference standard) and physician annotations. The HIVE model’s area under the receiver operating characteristic curve was evaluated using LLM-extracted vs physician-annotated features.

Results

Among 336 ED reports from patients with acute abdominal pain (median age 41, IQR 22‐62 years, 205/336, 61% female), 50% (167/336) had appendicitis. The LLM achieved weighted average accuracies of 0.910 (95% CI (0.018) with minimal prompts and 0.929 (95% CI ±0.016) with optimized prompts, vs 0.961 (95% CI ±0.012) and 0.951 (95% CI ±0.015) for physicians. Corresponding HIVE model area under the receiver operating characteristic curves were 0.871 (95% CI ±0.019) and 0.911 (95% CI ±0.014) with LLM inputs under the minimal and optimized prompts, compared to 0.917 (95% CI ±0.015) and 0.924 (95% CI ±0.018) for physician inputs.

Conclusions

A small locally deployable multilingual LLM can approach physician-level accuracy in extracting structured binary, multiclass, and multilabel clinical data from free-text Dutch ED reports, while preserving patient privacy, interpretability, and statistical transparency for downstream diagnostic modeling.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsEmergency Service, HospitalMachine LearningHumansRetrospective StudiesPredictive Learning ModelsNetherlandsROC Curve

Stichwörter der Autor:innen

electronic health recordsemergency medicinehealth informaticslarge language modelsmachine learningnatural language processingpredictive modeling

Autor:innen und Institutionen

  • Diagnostic Image Analysis Group, Medical Imaging Department, Radboud University Medical Center, Geert Grooteplein Zuid 10, Nijmegen, 6525 GA, The Netherlands, 31 614021323.
  • Laboratory of Clinical Chemistry and Hematology, Jeroen Bosch Ziekenhuis, Den Bosch, The Netherlands.
  • Psychiatry Department, Radboud University Medical Center, Nijmegen, The Netherlands.
  • Emergency Department, Jeroen Bosch Ziekenhuis, Den Bosch, The Netherlands.
  • Department of Health Technology and Services Research, Technical Medical Centre, University of Twente, Enschede, The Netherlands.
  • Laboratory of Clinical Chemistry and Laboratory Medicine, Dicoon BV, Canisius-Wilhelmina Ziekenhuis, Nijmegen, The Netherlands.

ORCID-Profile: Anoeska Schipper, Peter Belgers, Rory David O'Connor, Lieke van de Wouw, Luc Builtjes, Joeran S Bosma, Ron Kusters, Steef Kurstjens, Matthieu Rutten, Bram van Ginneken

Zitieren

APA

Schipper, A., Belgers, P., O'Connor, R. D., van de Wouw, L., Builtjes, L., Bosma, J. S., Kusters, R., Kurstjens, S., Rutten, M., & van Ginneken, B. (2026). Large Language Model Automated Extraction of Clinical Signs and Symptoms From Emergency Department Reports for Machine Learning Prediction Models: Development and Validation Study. JMIR Medical Informatics. https://doi.org/10.2196/81500

Vancouver

Schipper A, Belgers P, O'Connor RD, van de Wouw L, Builtjes L, Bosma JS, et al. Large Language Model Automated Extraction of Clinical Signs and Symptoms From Emergency Department Reports for Machine Learning Prediction Models: Development and Validation Study. JMIR Medical Informatics. 2026. doi:10.2196/81500

BibTeX
@article{Schipper2026,
  author = {Schipper, Anoeska and Belgers, Peter and O'Connor, Rory David and van de Wouw, Lieke and Builtjes, Luc and Bosma, Joeran S and Kusters, Ron and Kurstjens, Steef and Rutten, Matthieu and van Ginneken, Bram},
  title = {Large Language Model Automated Extraction of Clinical Signs and Symptoms From Emergency Department Reports for Machine Learning Prediction Models: Development and Validation Study},
  journal = {JMIR Medical Informatics},
  year = {2026},
  doi = {10.2196/81500}
}
Kostenlos starten →