Systematic Scoring Differences in Large Language Model–Based Clinical Benchmarking Reveal Evaluator-Dependent Divergence: Controlled Evaluation Study
JMIR AI · 26.10.2026 · 5 · e103962-e103962
DOI: 10.2196/103962 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR AI, online erschienen am 26.10.2026. Verfasst von Rasit Dinc und 1 weiteren Autor:innen. Beteiligte Einrichtung: INVAMED Institute for Medical Innovation, One World Trade Centre, 85th Floor, 285 Fulton Street, New York, NY, United States, 1 (669) 248-1675. Die Arbeit nennt 25 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language model (LLM) benchmarking in clinical medicine is increasingly reliant on LLM-based assessment frameworks; in these systems, one AI scores the outputs of another. This practice relies on the assumptions of evaluator neutrality and rubric consistency. These assumptions have not been empirically tested in clinical fields.
Objective
In this study, we aim to directly examine these assumptions by using the PARM (Planning-Action-Reflection-Memory) framework as a theoretical lens to characterize endovascular device selection and clinical reflective reasoning as a clinical field.
Methods
We developed a scoring rubric (C1-C6) with 40 clinical vignettes and 6 criteria across 5 endovascular fields (peripheral artery disease, abdominal aortic aneurysm, chronic venous insufficiency, coronary in-stent restenosis [ISR], and cerebrovascular/neuroendovascular; 8 vignettes per field). Four LLMs were evaluated as clinical decision support systems: GPT-4o (M1), Claude Sonnet 4.6 (M2), Gemini 1.5 Pro (M3), and Llama 3.3 70B (M4). Two LLMs served as independent raters with the same evaluation criteria: Claude Sonnet 4.6 (rater 1) and GPT-4o (rater 2). The rater identity was intentionally disclosed; the primary research question concerned rater scoring behavior rather than the absolute performance of the model. Human assessment was intentionally excluded because its inclusion would have shifted the study question from rater disagreement to rater accuracy, a distinct and secondary question. Interrater reliability was measured using intraclass correlation coefficient (ICC 2,1) two-way random effects modeling.
Results
The combined ICC(2,1) was 0.153 (95% CI 0.007‐0.295). In this exploratory study, the low ICC was not distributed randomly: the magnitude of score differences varied across models (M2 Δ=+5, M4 Δ=+31, M1 Δ=−46, and M3 Δ=−158 points). Scoring behavior analysis identified 2 intrinsically consistent but mutually inconsistent criterion-interpretation models. Rater 1’s scores were most consistent with the reasoning quality approach (partial credit to clinical competence independent of vignette context). Rater 2’s scores were most consistent with the context fidelity approach (near-zero scores for wrong vignette responses independent of reasoning quality). Contrary to the self-preference hypothesis, both raters placed the other higher-performing model above their own. Under rater 1, C6 (PARM reflection) was the lowest-scoring criterion across all 4 models, whereas rater 2 showed greater model-dependent variation in C6 scoring.
Conclusions
AI raters used in LLM-based clinical benchmarking do not behave as interchangeable scoring tools. Their scoring behavior is most consistent with model-specific criterion interpretation patterns that systematically produce different outcomes. The observed low interrater reliability requires interpretation rather than simply being ignored: it reflects structured, direction-specific disagreement concentrated in models with context-tracking failures, not random inconsistency. PARM reflection showed a notable yet evaluator-dependent scoring pattern that warranted further investigation, rather than being interpreted as an evaluator-independent deficit.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- INVAMED Institute for Medical Innovation, One World Trade Centre, 85th Floor, 285 Fulton Street, New York, NY, United States, 1 (669) 248-1675.
- Med-International UK Health Agency Ltd, Warwickshire, Nuneaton, United Kingdom.
Zitieren
APA
Dinc, R., & Ardic, N. (2026). Systematic Scoring Differences in Large Language Model–Based Clinical Benchmarking Reveal Evaluator-Dependent Divergence: Controlled Evaluation Study. JMIR AI, 5, e103962-e103962. https://doi.org/10.2196/103962
Vancouver
Dinc R, Ardic N. Systematic Scoring Differences in Large Language Model–Based Clinical Benchmarking Reveal Evaluator-Dependent Divergence: Controlled Evaluation Study. JMIR AI. 2026;5:e103962-e103962. doi:10.2196/103962
BibTeX
@article{Dinc2026,
author = {Dinc, Rasit and Ardic, Nurittin},
title = {Systematic Scoring Differences in Large Language Model–Based Clinical Benchmarking Reveal Evaluator-Dependent Divergence: Controlled Evaluation Study},
journal = {JMIR AI},
year = {2026},
volume = {5},
pages = {e103962--e103962},
doi = {10.2196/103962}
}
EUVIMED