EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Is Artificial Intelligence Ready for Emergency Department Triage? A Retrospective Evaluation of Multiple Large Language Models in 39,375 Patients at a University Emergency Department

Ioannis Nedos, Sofia-Chrysovalantou Zagalioti iD, Christos Kofos, Theoni Katsikidou, Dimitra Vellidou, Konstantinos Astrinakis, Ioannis Karagiannis, Panagiotis Giannakopoulos, Styliani Michaloudi, Aikaterini Apostolopoulou iD, Efstratios Karagiannidis iD, Barbara Fyntanidou iD

Journal of Clinical Medicine · 14.02.2026 · 15 (4) · 1512

DOI: 10.3390/jcm15041512 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Clinical Medicine, online erschienen am 14.02.2026. Verfasst von Ioannis Nedos und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Emergency Medicine, AHEPA University General Hospital, Aristotle University of Thessaloniki, 541 24 Thessaloniki, Greece. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) are increasingly proposed as clinical decision support tools. However, their reliability in the emergency department (ED) triage remains insufficiently validated. This study aimed to evaluate the performance and limitations of multiple LLMs in triage using a large retrospective dataset.

Methods

We conducted a retrospective analysis of 39,375 anonymized patient cases from the ED of AHEPA University General Hospital, Thessaloniki, Greece (June 2024–July 2025), extracted from the hospital’s electronic medical record system. All cases were triaged in real time according to the Emergency Severity Index (ESI) by 25 emergency physicians. In cases of uncertainty, a senior emergency physician was consulted. Seven LLMs (ChatGPT-5 Thinking, ChatGPT-5 Instant, Gemini 2.5, Qwen 3, Grok 4.0, Deep Seek v3.1, and Claude Sonnet 4) were evaluated against the physician-assigned ESI level (reference standard). Outcomes included triage score agreement (quadratic weighted kappa, κw), clinic referral accuracy and admission prediction. Subgroup analyses were performed by referral clinic and admission outcome. The study was conducted in accordance with TRIPOD-AI reporting guidelines.

Results

Model performance varied substantially. DeepSeek and Claude Sonnet 4 achieved the highest agreement with physician-assigned ESI (κw ≈ 0.467; raw accuracy: 61.7%). In contrast, GPT-5 Instant performed poorly across all evaluation metrics (κw = 0.176; 95% CI: 0.167–0.186). Claude Sonnet 4 demonstrated the best performance in clinic referral (67.1%; κ = 0.619) and admission prediction (κw ≈ 0.46). Subgroup analyses indicated higher performance in pediatric cases and organ-specific complaints, such as ophthalmology (up to 81% accuracy). LLMs also showed tendencies toward over- or under-triage.

Conclusions

Current LLMs demonstrate promising but inconsistent capability in triage. While selected models achieved moderate alignment with physician ESI decisions, none achieved strong agreement (κ > 0.80). LLMs are most suitable as supervised decision support tools, particularly in anatomically well-defined clinical scenarios, rather than as autonomous systems.

Schlagwörter

Stichwörter der Autor:innen

artificial intelligenceclinical decision supportemergency medicineemergency triagelarge language models

Autor:innen und Institutionen

  • Department of Emergency Medicine, AHEPA University General Hospital, Aristotle University of Thessaloniki, 541 24 Thessaloniki, Greece.

ORCID-Profile: Sofia-Chrysovalantou Zagalioti, Aikaterini Apostolopoulou, Efstratios Karagiannidis, Barbara Fyntanidou

Zitieren

APA

Nedos, I., Zagalioti, S. C., Kofos, C., Katsikidou, T., Vellidou, D., Astrinakis, K., Karagiannis, I., Giannakopoulos, P., Michaloudi, S., Apostolopoulou, A., Karagiannidis, E., & Fyntanidou, B. (2026). Is Artificial Intelligence Ready for Emergency Department Triage? A Retrospective Evaluation of Multiple Large Language Models in 39,375 Patients at a University Emergency Department. Journal of Clinical Medicine, 15(4), 1512. https://doi.org/10.3390/jcm15041512

Vancouver

Nedos I, Zagalioti SC, Kofos C, Katsikidou T, Vellidou D, Astrinakis K, et al. Is Artificial Intelligence Ready for Emergency Department Triage? A Retrospective Evaluation of Multiple Large Language Models in 39,375 Patients at a University Emergency Department. Journal of Clinical Medicine. 2026;15(4):1512. doi:10.3390/jcm15041512

BibTeX
@article{Nedos2026,
  author = {Nedos, Ioannis and Zagalioti, Sofia-Chrysovalantou and Kofos, Christos and Katsikidou, Theoni and Vellidou, Dimitra and Astrinakis, Konstantinos and Karagiannis, Ioannis and Giannakopoulos, Panagiotis and Michaloudi, Styliani and Apostolopoulou, Aikaterini and Karagiannidis, Efstratios and Fyntanidou, Barbara},
  title = {Is Artificial Intelligence Ready for Emergency Department Triage? A Retrospective Evaluation of Multiple Large Language Models in 39,375 Patients at a University Emergency Department},
  journal = {Journal of Clinical Medicine},
  year = {2026},
  volume = {15},
  number = {4},
  pages = {1512},
  doi = {10.3390/jcm15041512}
}
Kostenlos starten →