EUVIMEDEUVIMED entdecken
Crossref

Comparative evaluation of large language models for preliminary diagnosis and initial treatment decision-making in intensive care unit emergencies

Agah Abdullah Kahramanlar, Engin İhsan Turan

Journal of Medicine and Palliative Care · 25.10.2026 · 7 (5) · 985-992

DOI: 10.47582/jompac.1986801 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medicine and Palliative Care, online erschienen am 25.10.2026. Verfasst von Agah Abdullah Kahramanlar und 1 weiteren Autor:innen. Die Arbeit nennt 21 Literaturangaben. Gefördert von: None. The authors declared that this study has received no financial support..

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Aims

Timely and accurate decision-making is essential in intensive care unit (ICU) emergencies, where delays or errors in preliminary diagnosis and initial treatment may lead to adverse outcomes. Large language model–based Artificial Intelligence (AI) systems have shown promise in clinical reasoning tasks, yet their performance in high-acuity, time-sensitive ICU emergencies remains insufficiently studied.

Methods

This simulation-based observational study evaluated the performance of three large language model–based AI systems (Gemini, DeepSeek, and ChatGPT) in preliminary diagnosis and initial treatment decision-making across 84 simulated ICU emergency scenarios. Scenarios reflected real-world ICU practice and covered bleeding, cardiac, electrolyte imbalance, neurologic, respiratory, and other critical conditions. Each scenario included two multiple-choice questions (diagnosis and treatment). Responses from AI models were compared with the expert-derived reference standards established by two boardcertified anesthesiology specialists, with adjudication by a third expert in cases of disagreement. Model performance was assessed using accuracy, macro-averaged sensitivity, specificity, F1 score, and Cohen’s kappa coefficient. Subgroup analyses were performed by emergency type. The study was designed and reported in accordance with the TRIPOD-LLM statement.

Results

Overall diagnostic accuracy was 0.93 for Gemini, 0.92 for DeepSeek, and 0.65 for ChatGPT. Treatment decision accuracy was 0.98 for Gemini, 0.95 for DeepSeek, and 0.68 for ChatGPT. Agreement with the expert-derived reference standard was substantial for Gemini (κ=0.743 for diagnosis; κ=0.741 for treatment), substantial for DeepSeek diagnosis (κ=0.710) and moderate for DeepSeek treatment (κ=0.584), but low for ChatGPT (κ=0.144 and κ=0.085, respectively). Gemini and DeepSeek demonstrated consistently high subgroup accuracy across all emergency categories, including perfect performance in respiratory emergencies, whereas ChatGPT showed greater variability and lower accuracy across subgroups.

Conclusion

In simulated ICU emergency scenarios, Gemini and DeepSeek demonstrated high diagnostic and treatment decision accuracy, although agreement with the expert-derived reference standard varied by model and task, whereas ChatGPT showed lower and more variable performance. These findings suggest that selected large language models may have potential as decision-support tools in critical care settings; however, performance varies markedly between models. Further validation using real-world clinical data is required before clinical implementation.

Förderung

  • None. The authors declared that this study has received no financial support.

Zitieren

APA

Kahramanlar, A. A., & Turan, E. İ. (2026). Comparative evaluation of large language models for preliminary diagnosis and initial treatment decision-making in intensive care unit emergencies. Journal of Medicine and Palliative Care, 7(5), 985-992. https://doi.org/10.47582/jompac.1986801

Vancouver

Kahramanlar AA, Turan Eİ. Comparative evaluation of large language models for preliminary diagnosis and initial treatment decision-making in intensive care unit emergencies. Journal of Medicine and Palliative Care. 2026;7(5):985-992. doi:10.47582/jompac.1986801

BibTeX
@article{Kahramanlar2026,
  author = {Kahramanlar, Agah Abdullah and Turan, Engin İhsan},
  title = {Comparative evaluation of large language models for preliminary diagnosis and initial treatment decision-making in intensive care unit emergencies},
  journal = {Journal of Medicine and Palliative Care},
  year = {2026},
  volume = {7},
  number = {5},
  pages = {985--992},
  doi = {10.47582/jompac.1986801}
}
Kostenlos starten →