Large language models are unstable as autonomous medical evaluators across clinical tasks
Journal of the American Medical Informatics Association · 06.11.2026
DOI: 10.1093/jamia/ocag190 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 06.11.2026. Verfasst von Marcelo Carvalho E Silva und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Pediatrics, Faculdade de Medicina, Universidade de São Paulo (USP), São Paulo, SP 01246-903, Brazil. Die Arbeit nennt 23 Literaturangaben.
Abstract (Originalsprache)
Objective
Large language models (LLMs) are increasingly used as automated evaluators of medical text (“LLM-as-judge”). We tested whether LLM-evaluator performance validated on one medical task against a human reference standard remains stable on a structurally different task. Materials and methods We evaluated 10 LLMs (3 locally served open-source and 7 API models: Claude, GPT, DeepSeek) as autonomous judges on 3 tasks with human reference labels: LiveQA-Med answer quality (n = 300), PubMedQA research appraisal (n = 300), and Emergency Severity Index (ESI) triage on MIMIC-IV-Ext (n = 256). Each case was scored 3 times per model at temperature zero (25 680 judgments). Analyses used cluster-bootstrap confidence intervals, quadratic-weighted Cohen’s κ, and false-discovery-rate correction.
Results
Evaluator performance was unstable across tasks: no model ranked in the top 3 on all 3 tasks, and every model shifted by 3 or more rank positions between at least one pair of tasks. In the highest-stakes task, ESI level 1 (Resuscitation) cases were under-triaged in 89.1% of evaluations, in all 10 models (range 70.2%-96.9%). Best triage agreement only reached the human inter-rater range (Claude Opus quadratic-weighted κ = 0.60 [0.54, 0.66]). Locally served 7-27B models (no per-query charge) occupied the Pareto frontier on the text tasks but only its low-concordance end on triage.
Discussion
LLM-evaluator performance is unstable between medical tasks that differ in cognitive structure; performance validated on one task should not be assumed to transfer to another.
Conclusion
Task-specific validation against human reference labels is required before deployment, and emergency triage demands human oversight at the highest-acuity tier.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Pediatrics, Faculdade de Medicina, Universidade de São Paulo (USP), São Paulo, SP 01246-903, Brazil.
- Department of Anesthesiology, Hospital Unimed de Rondonópolis, Rondonópolis, MT 78750-793, Brazil.
- Department of Pediatrics, Hospital Sepaco, São Paulo, SP 04102-900, Brazil.
- Department of Internal Medicine, Irmandade Santa Casa de Piracicaba, Piracicaba, SP 13419-155, Brazil.
ORCID-Profile: Marcelo Carvalho E Silva
Zitieren
APA
Carvalho E Silva, M., Carvalho E Silva, F., de Paula Gomes Nunes de Castro, R., Gonçalves, A., & Figueiredo Delgado, A. (2026). Large language models are unstable as autonomous medical evaluators across clinical tasks. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag190
Vancouver
Carvalho E Silva M, Carvalho E Silva F, de Paula Gomes Nunes de Castro R, Gonçalves A, Figueiredo Delgado A. Large language models are unstable as autonomous medical evaluators across clinical tasks. Journal of the American Medical Informatics Association. 2026. doi:10.1093/jamia/ocag190
BibTeX
@article{CarvalhoESilva2026,
author = {Carvalho E Silva, Marcelo and Carvalho E Silva, Fernando and de Paula Gomes Nunes de Castro, Renata and Gonçalves, Alex and Figueiredo Delgado, Artur},
title = {Large language models are unstable as autonomous medical evaluators across clinical tasks},
journal = {Journal of the American Medical Informatics Association},
year = {2026},
doi = {10.1093/jamia/ocag190}
}
EUVIMED