Grounded large language models for diagnostic prediction in real-world emergency department settings
JAMIA Open · 02.09.2025 · 8 (5)
DOI: 10.1093/jamiaopen/ooaf119 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 21.10.2025. Verfasst von Alexandre Niset und 9 weiteren Autor:innen. Beteiligte Einrichtung: Emergency Department, Cliniques Universitaires Saint-Luc, Brussels, 1200, Belgium. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 67 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Objective
To evaluate predictive diagnostic performance of open- and closed-source large language models (LLMs) in emergency medicine, addressing the urgent need for innovative clinical decision support tools amid rising patient volumes and staffing shortages. Materials and Methods We generated 2370 AI-driven diagnostic predictions (Top-5 diagnoses from each of 6 model pipelines per patient), using data from 79 real-world emergency department cases collected consecutively during a 24-hour peak influx period at a tertiary care center. Pipelines combined open- and closed-source embedding models (text-embedding-ada-002, MXBAI) with foundational models (GPT-4, Llama3, and Qwen2) grounded via retrieval-augmented generation using emergency medicine textbooks. Models’ predictions were assessed against reference diagnoses established by expert consensus.
Results
All pipelines achieved comparable diagnostic match rates (62.03%-72.15%). Diagnostic performance was significantly influenced by case characteristics: match rates were notably higher for specific versus unspecific diagnoses (85.53% vs 31.41%, P < .001) and surgical versus medical cases (79.49% vs 56.25%, P < .001). Open-source models demonstrated markedly superior sourcing capabilities compared to GPT-4-based combinations (P < 1.4e-12), with MBXAI/Qwen2 pipeline achieving perfect citation verification.
Discussion
Diagnostic accuracy primarily depended on case characteristics rather than the choice of model pipeline, highlighting fundamental AI alignment challenges in clinical reasoning. Low performance in unspecific diagnoses underscores inherent complexities in clinical definitions rather than technological shortcomings alone.
Conclusion
Open-source LLM pipelines provide enhanced sourcing capabilities, crucial for transparent clinical decision-making and interpretability. Further research should expand knowledge bases to include hospital guidelines and regional epidemiology, while exploring on-premises solutions to better align with privacy regulations and clinical integration.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Emergency Department, Cliniques Universitaires Saint-Luc, Brussels, 1200, Belgium.
- Pediatric Intensive Care Unit, Cliniques Universitaires Saint-Luc, Brussels, 1200, Belgium.
- ScienseNew York, NY, 10004, United States.
- Emergency Department, CHU Helora-Jolimont, La Louvière, 7100, Belgium.
- Information and Communication Technologies, Electronics and Applied Mathematics (ICTEAM), UCLouvain, Louvain-la-Neuve, 1348, Belgium.
- Orthopedic Department, CHU UCL NamurYvoir, 5530, Belgium.
ORCID-Profile: Alexandre Niset
Zitieren
APA
Niset, A., Melot, I., Pireau, M., Englebert, A., Scius, N., Flament, J., El Hadwe, S., Al Barajraji, M., Thonon, H., & Barrit, S. (2025). Grounded large language models for diagnostic prediction in real-world emergency department settings. JAMIA Open, 8(5). https://doi.org/10.1093/jamiaopen/ooaf119
Vancouver
Niset A, Melot I, Pireau M, Englebert A, Scius N, Flament J, et al. Grounded large language models for diagnostic prediction in real-world emergency department settings. JAMIA Open. 2025;8(5). doi:10.1093/jamiaopen/ooaf119
BibTeX
@article{Niset2025,
author = {Niset, Alexandre and Melot, Ines and Pireau, Margaux and Englebert, Alexandre and Scius, Nathan and Flament, Julien and El Hadwe, Salim and Al Barajraji, Mejdeddine and Thonon, Henri and Barrit, Sami},
title = {Grounded large language models for diagnostic prediction in real-world emergency department settings},
journal = {JAMIA Open},
year = {2025},
volume = {8},
number = {5},
doi = {10.1093/jamiaopen/ooaf119}
}
EUVIMED