EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Prompt-Sensitive Decision Behavior of Large Language Models in Intensive Care Unit Mortality Prediction for Spontaneous Intracerebral Hemorrhage: Comparative Benchmarking Study

Jinn-Rung Kuo, Guan-Yu Chen, Xiao-Han Vivian Yap, Chao-Chien Li, Yung-De Kuo, Chung-Feng Liu

Journal of Medical Internet Research · 2026

DOI: 10.2196/29701 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of Medical Internet Research, 2026. Verfasst von Jinn-Rung Kuo und 5 weiteren Autor:innen. Beteiligte Einrichtung: School of Medicine, College of Medicine, National Sun Yat-Sen University, Kaohsiung, Taiwan. Hauptthemen laut MEDLINE: Cerebral Hemorrhage, Benchmarking, Intensive Care Units, Hospital Mortality. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 25 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) are increasingly being explored for clinical decision support. However, whether inference-only LLM outputs can be interpreted as reliable quantitative risk estimates in structured clinical prediction remains unclear.

Objective

This study aimed to evaluate the predictive performance and decision-making behavior of inference-only LLMs in a structured clinical prediction task and compare their outputs with those of an outcome-trained machine learning model.

Methods

We conducted a controlled benchmarking study using identical structured clinical inputs from patients admitted to intensive care units with spontaneous intracerebral hemorrhage. An outcome-trained extreme gradient boosting model was compared with predictions generated by a general-purpose LLM using four prompting strategies: zero-shot, few-shot, chain-of-thought, and combined few-shot plus chain-of-thought prompting. Performance was evaluated using discrimination metrics, threshold-dependent classification behavior, and concordance between Shapley Additive Explanations–derived feature importance rankings and LLM-derived feature prioritization. The independent testing cohort included 435 patients, of whom 86 (19.7%) experienced in-hospital mortality.

Results

The outcome-trained machine learning model demonstrated superior discriminative performance compared with all LLM-based approaches. LLM predictions achieved moderate discrimination but exhibited substantial variability in threshold-dependent classification behavior across prompting strategies. At a fixed probability threshold of 0.5, LLM approaches consistently demonstrated high sensitivity and lower specificity, whereas operating characteristics varied considerably when thresholds were optimized using the Youden index. The optimal thresholds for LLM-based approaches ranged from 0.74 to 0.88, compared with 0.1555 for the extreme gradient boosting model. Concordance between Shapley Additive Explanations–derived attribution and LLM-derived feature prioritization was modest, suggesting only partial alignment between empirically learned predictor structure and language-based reasoning patterns.

Conclusions

In this structured clinical prediction setting, inference-only LLM outputs demonstrated prompt-sensitive decision behavior despite moderate discriminative performance. These findings suggest that LLM-generated probability outputs should be interpreted cautiously when used for quantitative clinical risk estimation. A complementary framework integrating outcome-trained predictive models with LLM-assisted reasoning may provide a more reliable direction for future clinical decision support systems.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Cerebral HemorrhageBenchmarkingIntensive Care UnitsHospital MortalityDecision Support Systems, ClinicalHumansLarge Language ModelsMachine LearningBoosting Machine Learning AlgorithmsFemalePredictive Learning ModelsMale

Stichwörter der Autor:innen

SHAPShapley Additive Explanationsexplainable artificial intelligenceintracerebral hemorrhagelarge language modelsmachine learningprompt-sensitive behaviorstructured clinical prediction

Autor:innen und Institutionen

  • School of Medicine, College of Medicine, National Sun Yat-Sen University, Kaohsiung, Taiwan.
  • Department of Medical Research, Chi Mei Medical Center, Tainan, Taiwan.
  • Department of Neurosurgery, Chi Mei Medical Center, 901 Chung Hwa Road, Yung Kang District, Tainan, Taiwan, 886 6-281-2811 ext. 5.

Zitieren

APA

Kuo, J. R., Chen, G. Y., Yap, X. V., Li, C. C., Kuo, Y. D., & Liu, C. F. (2026). Prompt-Sensitive Decision Behavior of Large Language Models in Intensive Care Unit Mortality Prediction for Spontaneous Intracerebral Hemorrhage: Comparative Benchmarking Study. Journal of Medical Internet Research. https://doi.org/10.2196/29701

Vancouver

Kuo JR, Chen GY, Yap XV, Li CC, Kuo YD, Liu CF. Prompt-Sensitive Decision Behavior of Large Language Models in Intensive Care Unit Mortality Prediction for Spontaneous Intracerebral Hemorrhage: Comparative Benchmarking Study. Journal of Medical Internet Research. 2026. doi:10.2196/29701

BibTeX
@article{Kuo2026,
  author = {Kuo, Jinn-Rung and Chen, Guan-Yu and Yap, Xiao-Han Vivian and Li, Chao-Chien and Kuo, Yung-De and Liu, Chung-Feng},
  title = {Prompt-Sensitive Decision Behavior of Large Language Models in Intensive Care Unit Mortality Prediction for Spontaneous Intracerebral Hemorrhage: Comparative Benchmarking Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  doi = {10.2196/29701}
}
Kostenlos starten →