EUVIMEDObjevte EUVIMED
Open access · CC BY 4.0

Task-Difficulty-Aware Reliability Evaluation of Large Language Models for Clinical Decision Support

Ryan Cole, Benjamin Rhodes

Journal of Clinical Practice and Health Sciences · 2026-08-20 · 6 (2) · 87-100

DOI: 10.71465/jcphs73 ↗

At a glance

Journal article in Journal of Clinical Practice and Health Sciences, published online on 2026-08-20. Written by Ryan Cole and 1 co-authors. The full text is freely available (open access, CC BY 4.0).

AbstraktBibliografické údajeCiteRelated studies

Abstract (original language)

Model choice for clinical decision support (CDS) is increasingly justified by leaderboard accuracy on medical hallucination benchmarks, yet an accuracy figure is only interpretable relative to what a system with no medical knowledge could already obtain. We introduce DARE-CDS, a task-difficulty-aware reliability framework that models every benchmark task by a response curve whose pseudo-guessing parameter is measured directly from the released data instead of being assumed. Applying the framework to the full public Med-HALT corpus (59,254 items across seven tasks) and to the twelve checkpoints evaluated on it, we find that the gold label of the None-of-the-Above test is constant for all 18,866 items, that the Fake-Question scorer is satisfied by a constant abstention string, and that 1,989 of 4,916 PubMed items are unanswerable so that a constant "Unknown" reply scores exactly 40.46%, reproducing to the second decimal the best published memory-retrieval accuracy. A bank of 30 deterministic knowledge-free solvers bounds the reasoning floor at 32.95%. Consequently 34.97% of benchmark rows carry zero Fisher information, the effective test length is 26.1% of the nominal one, and only 2 of 84 model-task cells exceed the knowledge-free ceiling under Holm-corrected exact binomial tests. Difficulty-aware scoring reverses the leaderboard: the model that is top-ranked in 77.60% of random task mixtures under raw accuracy is top-ranked in 0.02% after correction. We further show that within-task item difficulty is not identifiable from knowledge-free responses, release a difficulty-controlled repair of the protocol that restores measurement capacity, and derive the item counts a medical reliability benchmark needs to support a defensible pre-deployment claim.

Cite

APA

Cole, R., & Rhodes, B. (2026). Task-Difficulty-Aware Reliability Evaluation of Large Language Models for Clinical Decision Support. Journal of Clinical Practice and Health Sciences, 6(2), 87-100. https://doi.org/10.71465/jcphs73

Vancouver

Cole R, Rhodes B. Task-Difficulty-Aware Reliability Evaluation of Large Language Models for Clinical Decision Support. Journal of Clinical Practice and Health Sciences. 2026;6(2):87-100. doi:10.71465/jcphs73

BibTeX
@article{Cole2026,
  author = {Cole, Ryan and Rhodes, Benjamin},
  title = {Task-Difficulty-Aware Reliability Evaluation of Large Language Models for Clinical Decision Support},
  journal = {Journal of Clinical Practice and Health Sciences},
  year = {2026},
  volume = {6},
  number = {2},
  pages = {87--100},
  doi = {10.71465/jcphs73}
}
Start for free →