Effect of evaluation prompt strategies on LLM-as-a-judge reliability in critical care
Anaesthesiology Intensive Therapy · 30.07.2026 · 58 (1) · 156-163
DOI: 10.5114/ait/221943 ↗Kurz gefasst
Fachartikel in der Zeitschrift Anaesthesiology Intensive Therapy, online erschienen am 30.07.2026. Verfasst von Jia-Yu Yan und 6 weiteren Autor:innen. Beteiligte Einrichtung: School of Medicine, College of Medicine, National Taiwan University, Taipei, Taiwan. Hauptthemen laut MEDLINE: Large Language Models, Critical Care, Artificial Intelligence. Die Arbeit nennt 29 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Introduction
Large language model (LLM)-as-a-judge systems offer scalable evaluation of artificial intelligence (AI)-generated clinical outputs, yet their susceptibility to prompt variability raises concerns regarding reproducibility and alignment with expert judgement. This study examined whether evaluation prompt strategies influence scoring patterns and concordance with clinical raters in critical care. Material and methods This post-hoc analysis used 90 structured clinical reports generated in a prior study using an XGBoost ICU mortality prediction model trained on the MIMIC-IV database. GPT-4o (Azure AI, version 2024-11-20) produced structured interpretations from risk estimates and SHAP attributions. These outputs were evaluated using the IMPACT framework under three evaluation prompt strategies: baseline (E1), top-down decremental (E2), and bottom-up incremental (E3). Agreement between clinician ratings and the automated o3-mini evaluator (Azure AI, version 2025-01-31) was assessed using intraclass correlation coefficients (ICC), with strategy comparisons by Fisher’s z-transformation. Score deviations were examined with repeated-measures ANOVA.
Results
Mean IMPACT scores were 79.9 (SD 9.9) for E1, 83.3 (SD 9.6) for E2, 78.7 (SD 9.1) for E3, and 78.6 (SD 8.9) for clinicians. All strategies demonstrated substantial agreement (ICC > 0.80). E2 showed significantly lower agreement with clinicians (ICC = 0.82) than E1 and E3 (both ICC = 0.94, p < 0.001). Score deviations differed significantly across strategies (p < 0.001), with E3 showing the smallest mean deviation (0.1) and E2 the largest (4.7).
Conclusions
Prompt design meaningfully affects both IMPACT scoring patterns and the reliability of LLM-based evaluators. Bottom-up incremental scoring showed the closest alignment with human assessment, underscoring the need for standardised prompt architectures in clinical AI evaluation.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Medicine, College of Medicine, National Taiwan University, Taipei, Taiwan.
- Department of Anesthesiology, Far Eastern Memorial Hospital, New Taipei City, Taiwan.
- Department of Anaesthesiology, National Taiwan University Hospital, Taipei, Taiwan.
- Department of Medicine, National Taiwan University Hospital, Taipei, Taiwan.
Zitieren
APA
Yan, J. Y., Chan, W. S., Chiu, C. T., Chao, A., Hsiao, P. N., Ku, S. C., & Yeh, Y. C. (2026). Effect of evaluation prompt strategies on LLM-as-a-judge reliability in critical care. Anaesthesiology Intensive Therapy, 58(1), 156-163. https://doi.org/10.5114/ait/221943
Vancouver
Yan JY, Chan WS, Chiu CT, Chao A, Hsiao PN, Ku SC, et al. Effect of evaluation prompt strategies on LLM-as-a-judge reliability in critical care. Anaesthesiology Intensive Therapy. 2026;58(1):156-163. doi:10.5114/ait/221943
BibTeX
@article{Yan2026,
author = {Yan, Jia-Yu and Chan, Wing-Sum and Chiu, Ching-Tang and Chao, Anne and Hsiao, Po-Ni and Ku, Shih-Chi and Yeh, Yu-Chang},
title = {Effect of evaluation prompt strategies on LLM-as-a-judge reliability in critical care},
journal = {Anaesthesiology Intensive Therapy},
year = {2026},
volume = {58},
number = {1},
pages = {156--163},
doi = {10.5114/ait/221943}
}
EUVIMED