EUVIMEDEUVIMED entdecken
Open Access

Effect of evaluation prompt strategies on LLM-as-a-judge reliability in critical care

Jia-Yu Yan, Wing-Sum Chan, Ching-Tang Chiu, Anne Chao, Po-Ni Hsiao, Shih-Chi Ku, Yu-Chang Yeh

Anaesthesiology Intensive Therapy · 30.07.2026 · 58 (1) · 156-163

DOI: 10.5114/ait/221943 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Anaesthesiology Intensive Therapy, online erschienen am 30.07.2026. Verfasst von Jia-Yu Yan und 6 weiteren Autor:innen. Beteiligte Einrichtung: School of Medicine, College of Medicine, National Taiwan University, Taipei, Taiwan. Hauptthemen laut MEDLINE: Large Language Models, Critical Care, Artificial Intelligence. Die Arbeit nennt 29 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

Large language model (LLM)-as-a-judge systems offer scalable evaluation of artificial intelligence (AI)-generated clinical outputs, yet their susceptibility to prompt variability raises concerns regarding reproducibility and alignment with expert judgement. This study examined whether evaluation prompt strategies influence scoring patterns and concordance with clinical raters in critical care. Material and methods This post-hoc analysis used 90 structured clinical reports generated in a prior study using an XGBoost ICU mortality prediction model trained on the MIMIC-IV database. GPT-4o (Azure AI, version 2024-11-20) produced structured interpretations from risk estimates and SHAP attributions. These outputs were evaluated using the IMPACT framework under three evaluation prompt strategies: baseline (E1), top-down decremental (E2), and bottom-up incremental (E3). Agreement between clinician ratings and the automated o3-mini evaluator (Azure AI, version 2025-01-31) was assessed using intraclass correlation coefficients (ICC), with strategy comparisons by Fisher’s z-transformation. Score deviations were examined with repeated-measures ANOVA.

Results

Mean IMPACT scores were 79.9 (SD 9.9) for E1, 83.3 (SD 9.6) for E2, 78.7 (SD 9.1) for E3, and 78.6 (SD 8.9) for clinicians. All strategies demonstrated substantial agreement (ICC > 0.80). E2 showed significantly lower agreement with clinicians (ICC = 0.82) than E1 and E3 (both ICC = 0.94, p < 0.001). Score deviations differed significantly across strategies (p < 0.001), with E3 showing the smallest mean deviation (0.1) and E2 the largest (4.7).

Conclusions

Prompt design meaningfully affects both IMPACT scoring patterns and the reliability of LLM-based evaluators. Bottom-up incremental scoring showed the closest alignment with human assessment, underscoring the need for standardised prompt architectures in clinical AI evaluation.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsCritical CareArtificial IntelligenceHumansReproducibility of ResultsGenerative Artificial Intelligence

Stichwörter der Autor:innen

critical caregenerative artificial intelligencelarge language modelprompt engineering

Autor:innen und Institutionen

  • School of Medicine, College of Medicine, National Taiwan University, Taipei, Taiwan.
  • Department of Anesthesiology, Far Eastern Memorial Hospital, New Taipei City, Taiwan.
  • Department of Anaesthesiology, National Taiwan University Hospital, Taipei, Taiwan.
  • Department of Medicine, National Taiwan University Hospital, Taipei, Taiwan.

Zitieren

APA

Yan, J. Y., Chan, W. S., Chiu, C. T., Chao, A., Hsiao, P. N., Ku, S. C., & Yeh, Y. C. (2026). Effect of evaluation prompt strategies on LLM-as-a-judge reliability in critical care. Anaesthesiology Intensive Therapy, 58(1), 156-163. https://doi.org/10.5114/ait/221943

Vancouver

Yan JY, Chan WS, Chiu CT, Chao A, Hsiao PN, Ku SC, et al. Effect of evaluation prompt strategies on LLM-as-a-judge reliability in critical care. Anaesthesiology Intensive Therapy. 2026;58(1):156-163. doi:10.5114/ait/221943

BibTeX
@article{Yan2026,
  author = {Yan, Jia-Yu and Chan, Wing-Sum and Chiu, Ching-Tang and Chao, Anne and Hsiao, Po-Ni and Ku, Shih-Chi and Yeh, Yu-Chang},
  title = {Effect of evaluation prompt strategies on LLM-as-a-judge reliability in critical care},
  journal = {Anaesthesiology Intensive Therapy},
  year = {2026},
  volume = {58},
  number = {1},
  pages = {156--163},
  doi = {10.5114/ait/221943}
}
Kostenlos starten →