Task-Difficulty-Aware Reliability Evaluation of Large Language Models for Clinical Decision Support
Journal of Clinical Practice and Health Sciences · 20.08.2026 · 6 (2) · 87-100
DOI: 10.71465/jcphs73 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Clinical Practice and Health Sciences, online erschienen am 20.08.2026. Verfasst von Ryan Cole und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Model choice for clinical decision support (CDS) is increasingly justified by leaderboard accuracy on medical hallucination benchmarks, yet an accuracy figure is only interpretable relative to what a system with no medical knowledge could already obtain. We introduce DARE-CDS, a task-difficulty-aware reliability framework that models every benchmark task by a response curve whose pseudo-guessing parameter is measured directly from the released data instead of being assumed. Applying the framework to the full public Med-HALT corpus (59,254 items across seven tasks) and to the twelve checkpoints evaluated on it, we find that the gold label of the None-of-the-Above test is constant for all 18,866 items, that the Fake-Question scorer is satisfied by a constant abstention string, and that 1,989 of 4,916 PubMed items are unanswerable so that a constant "Unknown" reply scores exactly 40.46%, reproducing to the second decimal the best published memory-retrieval accuracy. A bank of 30 deterministic knowledge-free solvers bounds the reasoning floor at 32.95%. Consequently 34.97% of benchmark rows carry zero Fisher information, the effective test length is 26.1% of the nominal one, and only 2 of 84 model-task cells exceed the knowledge-free ceiling under Holm-corrected exact binomial tests. Difficulty-aware scoring reverses the leaderboard: the model that is top-ranked in 77.60% of random task mixtures under raw accuracy is top-ranked in 0.02% after correction. We further show that within-task item difficulty is not identifiable from knowledge-free responses, release a difficulty-controlled repair of the protocol that restores measurement capacity, and derive the item counts a medical reliability benchmark needs to support a defensible pre-deployment claim.
Zitieren
APA
Cole, R., & Rhodes, B. (2026). Task-Difficulty-Aware Reliability Evaluation of Large Language Models for Clinical Decision Support. Journal of Clinical Practice and Health Sciences, 6(2), 87-100. https://doi.org/10.71465/jcphs73
Vancouver
Cole R, Rhodes B. Task-Difficulty-Aware Reliability Evaluation of Large Language Models for Clinical Decision Support. Journal of Clinical Practice and Health Sciences. 2026;6(2):87-100. doi:10.71465/jcphs73
BibTeX
@article{Cole2026,
author = {Cole, Ryan and Rhodes, Benjamin},
title = {Task-Difficulty-Aware Reliability Evaluation of Large Language Models for Clinical Decision Support},
journal = {Journal of Clinical Practice and Health Sciences},
year = {2026},
volume = {6},
number = {2},
pages = {87--100},
doi = {10.71465/jcphs73}
}
EUVIMED