Comparative Evaluation of Advanced Reasoning Models for Clinical Decision Support in Urology
Urologia Internationalis · 23.03.2026 · 1-10
DOI: 10.1159/000551610 ↗Kurz gefasst
Fachartikel in der Zeitschrift Urologia Internationalis, online erschienen am 23.03.2026. Verfasst von Ning He und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Urology, Hanzhong Central Hospital, Hanzhong, China. Die Arbeit nennt 7 Literaturangaben.
Abstract (Originalsprache)
Introduction
Advanced reasoning large language models are increasingly being explored for clinical decision support, but evidence regarding their performance in urology-specific tasks and their comparative standing against human respondents remains limited. This study compared five advanced reasoning models on urology-related multiple-choice questions from the MedQA dataset and benchmarked their performance against medical students and experienced urologists in terms of accuracy, response efficiency, and agreement patterns.
Methods
We extracted 434 urology-relevant items and evaluated five models – DeepSeek-R1, ChatGPT O4-mini, Gemini 2.5 Pro, Claude 3.7 Sonnet, and Grok 3 – using a standardized prompt. Accuracy was computed against reference answers; API response times and connection failures were recorded. In addition, 20 senior medical students and 20 experienced urologists answered subsets of the same item bank using a balanced block design; group-level majority-vote answers were used as human baselines. Statistical analyses included Cochran’s Q and McNemar tests (artificial intelligence [AI]-only accuracy), a logistic generalized linear mixed-effects model with urologists as the reference (model-adjusted accuracy), Fleiss’ kappa and Cohen’s kappa (agreement), and Friedman and Wilcoxon signed-rank tests (response time).
Results
Across the AI-only comparison, all models achieved high accuracy (86.9–93.3%), with DeepSeek-R1, ChatGPT O4-mini, and Gemini 2.5 Pro outperforming Claude 3.7 Sonnet and Grok 3. In the model-adjusted analysis, all five AI models showed significantly higher odds of correct answers than experienced urologists (all p < 0.001, Dunnett-adjusted), while medical students did not differ significantly from urologists. ChatGPT O4-mini had the shortest median API response time (5.03 s), whereas group-level median task completion times were 15.87 s for students and 17.57 s for urologists; Grok 3 was the slowest among AI models (27.62 s). Connection failure rates were 0% for ChatGPT O4-mini, Gemini 2.5 Pro, and Claude 3.7 Sonnet; 1.6% for DeepSeek-R1; and 2.8% for Grok 3. Agreement across the five AI models and the two human majority-vote baselines was moderate-to-substantial (Fleiss’ κ = 0.685, p < 0.001).
Conclusion
Modern reasoning models achieve strong accuracy and efficiency on urology-focused benchmark questions, supporting their potential role as useful clinical assistants when implemented with appropriate human oversight. ChatGPT O4-mini’s rapid latency further underscores its suitability for time-sensitive workflows, while model-adjusted analyses indicate its consistently superior accuracy relative to experienced urologists within this standardized assessment format.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Urology, Hanzhong Central Hospital, Hanzhong, China.
- Department of Urology, Hanzhong Central Hospital, Hanzhong, China, 1352057334@qq.com.
Zitieren
APA
He, N., Wang, Y., Hu, X., Zhang, Z., Bai, Y., Wang, J., Zheng, N., & Shao, L. (2026). Comparative Evaluation of Advanced Reasoning Models for Clinical Decision Support in Urology. Urologia Internationalis, 1-10. https://doi.org/10.1159/000551610
Vancouver
He N, Wang Y, Hu X, Zhang Z, Bai Y, Wang J, et al. Comparative Evaluation of Advanced Reasoning Models for Clinical Decision Support in Urology. Urologia Internationalis. 2026:1-10. doi:10.1159/000551610
BibTeX
@article{He2026,
author = {He, Ning and Wang, Yang and Hu, Xinhong and Zhang, Zhanhong and Bai, Yangkai and Wang, Jianfu and Zheng, Nan and Shao, Linhai},
title = {Comparative Evaluation of Advanced Reasoning Models for Clinical Decision Support in Urology},
journal = {Urologia Internationalis},
year = {2026},
pages = {1--10},
doi = {10.1159/000551610}
}
EUVIMED