EUVIMEDEUVIMED entdecken
Crossref

Comparative Evaluation of Advanced Reasoning Models for Clinical Decision Support in Urology

Ning He, Yang Wang, Xinhong Hu, Zhanhong Zhang, Yangkai Bai, Jianfu Wang, Nan Zheng, Linhai Shao

Urologia Internationalis · 23.03.2026 · 1-10

DOI: 10.1159/000551610 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Urologia Internationalis, online erschienen am 23.03.2026. Verfasst von Ning He und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Urology, Hanzhong Central Hospital, Hanzhong, China. Die Arbeit nennt 7 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

Advanced reasoning large language models are increasingly being explored for clinical decision support, but evidence regarding their performance in urology-specific tasks and their comparative standing against human respondents remains limited. This study compared five advanced reasoning models on urology-related multiple-choice questions from the MedQA dataset and benchmarked their performance against medical students and experienced urologists in terms of accuracy, response efficiency, and agreement patterns.

Methods

We extracted 434 urology-relevant items and evaluated five models – DeepSeek-R1, ChatGPT O4-mini, Gemini 2.5 Pro, Claude 3.7 Sonnet, and Grok 3 – using a standardized prompt. Accuracy was computed against reference answers; API response times and connection failures were recorded. In addition, 20 senior medical students and 20 experienced urologists answered subsets of the same item bank using a balanced block design; group-level majority-vote answers were used as human baselines. Statistical analyses included Cochran’s Q and McNemar tests (artificial intelligence [AI]-only accuracy), a logistic generalized linear mixed-effects model with urologists as the reference (model-adjusted accuracy), Fleiss’ kappa and Cohen’s kappa (agreement), and Friedman and Wilcoxon signed-rank tests (response time).

Results

Across the AI-only comparison, all models achieved high accuracy (86.9–93.3%), with DeepSeek-R1, ChatGPT O4-mini, and Gemini 2.5 Pro outperforming Claude 3.7 Sonnet and Grok 3. In the model-adjusted analysis, all five AI models showed significantly higher odds of correct answers than experienced urologists (all p < 0.001, Dunnett-adjusted), while medical students did not differ significantly from urologists. ChatGPT O4-mini had the shortest median API response time (5.03 s), whereas group-level median task completion times were 15.87 s for students and 17.57 s for urologists; Grok 3 was the slowest among AI models (27.62 s). Connection failure rates were 0% for ChatGPT O4-mini, Gemini 2.5 Pro, and Claude 3.7 Sonnet; 1.6% for DeepSeek-R1; and 2.8% for Grok 3. Agreement across the five AI models and the two human majority-vote baselines was moderate-to-substantial (Fleiss’ κ = 0.685, p < 0.001).

Conclusion

Modern reasoning models achieve strong accuracy and efficiency on urology-focused benchmark questions, supporting their potential role as useful clinical assistants when implemented with appropriate human oversight. ChatGPT O4-mini’s rapid latency further underscores its suitability for time-sensitive workflows, while model-adjusted analyses indicate its consistently superior accuracy relative to experienced urologists within this standardized assessment format.

Schlagwörter

Stichwörter der Autor:innen

Artificial intelligenceClinical decision supportLarge language modelsMedical educationUrology

Autor:innen und Institutionen

  • Department of Urology, Hanzhong Central Hospital, Hanzhong, China.
  • Department of Urology, Hanzhong Central Hospital, Hanzhong, China, 1352057334@qq.com.

Zitieren

APA

He, N., Wang, Y., Hu, X., Zhang, Z., Bai, Y., Wang, J., Zheng, N., & Shao, L. (2026). Comparative Evaluation of Advanced Reasoning Models for Clinical Decision Support in Urology. Urologia Internationalis, 1-10. https://doi.org/10.1159/000551610

Vancouver

He N, Wang Y, Hu X, Zhang Z, Bai Y, Wang J, et al. Comparative Evaluation of Advanced Reasoning Models for Clinical Decision Support in Urology. Urologia Internationalis. 2026:1-10. doi:10.1159/000551610

BibTeX
@article{He2026,
  author = {He, Ning and Wang, Yang and Hu, Xinhong and Zhang, Zhanhong and Bai, Yangkai and Wang, Jianfu and Zheng, Nan and Shao, Linhai},
  title = {Comparative Evaluation of Advanced Reasoning Models for Clinical Decision Support in Urology},
  journal = {Urologia Internationalis},
  year = {2026},
  pages = {1--10},
  doi = {10.1159/000551610}
}
Kostenlos starten →