Is Artificial Intelligence Ready for Emergency Department Triage? A Retrospective Evaluation of Multiple Large Language Models in 39,375 Patients at a University Emergency Department
Journal of Clinical Medicine · 14.02.2026 · 15 (4) · 1512
DOI: 10.3390/jcm15041512 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Clinical Medicine, online erschienen am 14.02.2026. Verfasst von Ioannis Nedos und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Emergency Medicine, AHEPA University General Hospital, Aristotle University of Thessaloniki, 541 24 Thessaloniki, Greece. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
Large language models (LLMs) are increasingly proposed as clinical decision support tools. However, their reliability in the emergency department (ED) triage remains insufficiently validated. This study aimed to evaluate the performance and limitations of multiple LLMs in triage using a large retrospective dataset.
Methods
We conducted a retrospective analysis of 39,375 anonymized patient cases from the ED of AHEPA University General Hospital, Thessaloniki, Greece (June 2024–July 2025), extracted from the hospital’s electronic medical record system. All cases were triaged in real time according to the Emergency Severity Index (ESI) by 25 emergency physicians. In cases of uncertainty, a senior emergency physician was consulted. Seven LLMs (ChatGPT-5 Thinking, ChatGPT-5 Instant, Gemini 2.5, Qwen 3, Grok 4.0, Deep Seek v3.1, and Claude Sonnet 4) were evaluated against the physician-assigned ESI level (reference standard). Outcomes included triage score agreement (quadratic weighted kappa, κw), clinic referral accuracy and admission prediction. Subgroup analyses were performed by referral clinic and admission outcome. The study was conducted in accordance with TRIPOD-AI reporting guidelines.
Results
Model performance varied substantially. DeepSeek and Claude Sonnet 4 achieved the highest agreement with physician-assigned ESI (κw ≈ 0.467; raw accuracy: 61.7%). In contrast, GPT-5 Instant performed poorly across all evaluation metrics (κw = 0.176; 95% CI: 0.167–0.186). Claude Sonnet 4 demonstrated the best performance in clinic referral (67.1%; κ = 0.619) and admission prediction (κw ≈ 0.46). Subgroup analyses indicated higher performance in pediatric cases and organ-specific complaints, such as ophthalmology (up to 81% accuracy). LLMs also showed tendencies toward over- or under-triage.
Conclusions
Current LLMs demonstrate promising but inconsistent capability in triage. While selected models achieved moderate alignment with physician ESI decisions, none achieved strong agreement (κ > 0.80). LLMs are most suitable as supervised decision support tools, particularly in anatomically well-defined clinical scenarios, rather than as autonomous systems.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Emergency Medicine, AHEPA University General Hospital, Aristotle University of Thessaloniki, 541 24 Thessaloniki, Greece.
ORCID-Profile: Sofia-Chrysovalantou Zagalioti, Aikaterini Apostolopoulou, Efstratios Karagiannidis, Barbara Fyntanidou
Zitieren
APA
Nedos, I., Zagalioti, S. C., Kofos, C., Katsikidou, T., Vellidou, D., Astrinakis, K., Karagiannis, I., Giannakopoulos, P., Michaloudi, S., Apostolopoulou, A., Karagiannidis, E., & Fyntanidou, B. (2026). Is Artificial Intelligence Ready for Emergency Department Triage? A Retrospective Evaluation of Multiple Large Language Models in 39,375 Patients at a University Emergency Department. Journal of Clinical Medicine, 15(4), 1512. https://doi.org/10.3390/jcm15041512
Vancouver
Nedos I, Zagalioti SC, Kofos C, Katsikidou T, Vellidou D, Astrinakis K, et al. Is Artificial Intelligence Ready for Emergency Department Triage? A Retrospective Evaluation of Multiple Large Language Models in 39,375 Patients at a University Emergency Department. Journal of Clinical Medicine. 2026;15(4):1512. doi:10.3390/jcm15041512
BibTeX
@article{Nedos2026,
author = {Nedos, Ioannis and Zagalioti, Sofia-Chrysovalantou and Kofos, Christos and Katsikidou, Theoni and Vellidou, Dimitra and Astrinakis, Konstantinos and Karagiannis, Ioannis and Giannakopoulos, Panagiotis and Michaloudi, Styliani and Apostolopoulou, Aikaterini and Karagiannidis, Efstratios and Fyntanidou, Barbara},
title = {Is Artificial Intelligence Ready for Emergency Department Triage? A Retrospective Evaluation of Multiple Large Language Models in 39,375 Patients at a University Emergency Department},
journal = {Journal of Clinical Medicine},
year = {2026},
volume = {15},
number = {4},
pages = {1512},
doi = {10.3390/jcm15041512}
}
EUVIMED