EUVIMEDEUVIMED entdecken
Open Access

Challenges of Single-Reference Benchmarks for Clinical Reasoning in Large Language Model–Based Decision Support

Jiawei Zhou, Hanqing He, Weijie Yu, Zihan Zhong, Qiang Ju, Shuhang Wang, Xiaochuan Wang, Ning Li

Journal of Medical Internet Research · 13.11.2026 · 28 · e99019-e99019

DOI: 10.2196/99019 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 13.11.2026. Verfasst von Jiawei Zhou und 7 weiteren Autor:innen. Beteiligte Einrichtung: Cancer Hospital of Chinese Academy of Medical Sciences, Panjiayuan Nanli 17, Beijing, 100021, China, 86 0316-591-8787. Hauptthemen laut MEDLINE: Benchmarking, Decision Support Systems, Clinical, Clinical Reasoning. Die Arbeit nennt 16 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This viewpoint addresses developers and evaluators of large language model (LLM)–based clinical decision support systems. Although recent benchmarks increasingly evaluate multiple clinical reasoning domains, many still rely substantially on reference decision agreement—whether model outputs conform to predefined decisions. Our aim is to clarify when such agreement remains useful, when it becomes insufficient for complex or preference-sensitive decisions, and how it can be complemented by reasoning-centered assessment. Clinical decision-making often depends on structured reasoning under uncertainty, integrating probabilistic evidence, contextual constraints, and patient values. Using oncology as an illustrative domain, we argue that some decision variability is a structural feature of clinical reasoning rather than noise or error. Used alone in such settings, single-reference benchmarks may conflate reasonable divergence with incorrectness and reward answer imitation over transparent reasoning. We conclude that reference decision agreement should be retained for tasks with stable ground truth but supplemented by evidence grounding, reasoning quality, patient-centeredness, and safety checks for open-ended clinical decision support. This multidimensional approach is intended to support safer and more clinically meaningful evaluation.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

BenchmarkingDecision Support Systems, ClinicalClinical ReasoningLarge Language ModelsHumans

Stichwörter der Autor:innen

benchmarkclinical decisionlarge language modeloncologyreasoning

Autor:innen und Institutionen

  • Cancer Hospital of Chinese Academy of Medical Sciences, Panjiayuan Nanli 17, Beijing, 100021, China, 86 0316-591-8787.
  • Baichuan Inc, Beijing, China.

Zitieren

APA

Zhou, J., He, H., Yu, W., Zhong, Z., Ju, Q., Wang, S., Wang, X., & Li, N. (2026). Challenges of Single-Reference Benchmarks for Clinical Reasoning in Large Language Model–Based Decision Support. Journal of Medical Internet Research, 28, e99019-e99019. https://doi.org/10.2196/99019

Vancouver

Zhou J, He H, Yu W, Zhong Z, Ju Q, Wang S, et al. Challenges of Single-Reference Benchmarks for Clinical Reasoning in Large Language Model–Based Decision Support. Journal of Medical Internet Research. 2026;28:e99019-e99019. doi:10.2196/99019

BibTeX
@article{Zhou2026,
  author = {Zhou, Jiawei and He, Hanqing and Yu, Weijie and Zhong, Zihan and Ju, Qiang and Wang, Shuhang and Wang, Xiaochuan and Li, Ning},
  title = {Challenges of Single-Reference Benchmarks for Clinical Reasoning in Large Language Model–Based Decision Support},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e99019--e99019},
  doi = {10.2196/99019}
}
Kostenlos starten →