Evaluation of large language models for clinical decision support in renal cell carcinoma: a multidisciplinary Delphi study
Journal of Comparative Effectiveness Research · 2026-11 · 15 (11s)
DOI: 10.57264/cer-2026-0082 ↗Kurz gefasst
Delphi-Studie in der Zeitschrift Journal of Comparative Effectiveness Research, 2026-11. Verfasst von Sergio Fernández-Pello Montes und 11 weiteren Autor:innen. Beteiligte Einrichtung: Urology Department, Hospital Universitario de Cabueñes, Gijón, Spain. Hauptthemen laut MEDLINE: Large Language Models, Carcinoma, Renal Cell, Kidney Neoplasms, Decision Support Systems, Clinical. Die Arbeit nennt 27 Literaturangaben.
Abstract (Originalsprache)
Aim
Large language models (LLMs) may facilitate the interpretation of clinical guideline information, but their performance in renal cell carcinoma (RCC) remains insufficiently characterized. We evaluated agreement between a multidisciplinary expert panel and the responses generated by Claude, Gemini and ChatGPT to RCC guideline-based questions. Materials & methods: A multidisciplinary tumor board developed 25 clinical questions based on the 2025 European Association of Urology RCC guidelines. Guideline figures and tables were converted into structured, plain-language textual summaries and supplied to the models. Each LLM generated one response per question using direct prompting. An 11-member multidisciplinary expert panel evaluated the responses in a two-round modified Delphi process using a 9-point scale and predefined consensus criteria.
Results
Consensus in agreement was reached for 24/25 Gemini responses (96%) and 23/25 responses (92%) for both Claude and ChatGPT. No response reached consensus in disagreement. The proportion of responses with a median score ≥7 was 25/25 (100%) for Gemini, 24/25 (96%) for Claude and 23/25 (92%) for ChatGPT. No statistically significant differences were detected among the models overall or in pairwise comparisons. Responses requiring further discussion or failing to reach consensus mainly involved clinically uncertain or individualized scenarios. Panelists identified occasional factual inaccuracies, omitted options and limited clinical nuance.
Conclusion
When provided with structured, guideline-derived material, selected LLMs generated responses that frequently aligned with multidisciplinary expert consensus. These findings support their potential for guideline interpretation but do not demonstrate independent clinical reasoning or real-world decision-making performance. Expert oversight remains essential, particularly in complex scenarios.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Urology Department, Hospital Universitario de Cabueñes, Gijón, Spain.
- Urology & Nephrology Department, Hospital Universitario Virgen del Rocío, Instituto Biomédico de Sevilla (Ibis), Universidad de Sevilla, Sevilla, Spain.
- Urology Department, Hospital Universitario Torrecárdenas, Almeria, Spain.
- Pharmacy Department, Hospital Universitario Miguel Servet, Zaragoza, Spain.
- Pathology Department, Fundación Puigvert, Barcelona, Spain.
- Radiotherapy Department, Hospital Central de la Defensa Gomez Ulla, Madrid, Spain.
Zitieren
APA
Montes, S. F., García, I. O., Martín, R. B., Agustín Ferrández, M. J., Arrea, F. A., Esteban, E. C., Imbroda, B. H., Mir Maresma, M. C., Nicolau, C., Perdigó, M. S., Velasco Oria de Rueda, G., & Borque-Fernando, Á. (2026). Evaluation of large language models for clinical decision support in renal cell carcinoma: a multidisciplinary Delphi study. Journal of Comparative Effectiveness Research, 15(11s). https://doi.org/10.57264/cer-2026-0082
Vancouver
Montes SF, García IO, Martín RB, Agustín Ferrández MJ, Arrea FA, Esteban EC, et al. Evaluation of large language models for clinical decision support in renal cell carcinoma: a multidisciplinary Delphi study. Journal of Comparative Effectiveness Research. 2026;15(11s). doi:10.57264/cer-2026-0082
BibTeX
@article{Montes2026,
author = {Montes, Sergio Fernández-Pello and García, Ignacio Osman and Martín, Rocio Barrabino and Agustín Ferrández, María José and Arrea, Ferrán Algaba and Esteban, Eliseo Carrasco and Imbroda, Bernardo Herrera and Mir Maresma, María Carmen and Nicolau, Carlos and Perdigó, Marc Simó and Velasco Oria de Rueda, Guillermo de and Borque-Fernando, Ángel},
title = {Evaluation of large language models for clinical decision support in renal cell carcinoma: a multidisciplinary Delphi study},
journal = {Journal of Comparative Effectiveness Research},
year = {2026},
volume = {15},
number = {11s},
doi = {10.57264/cer-2026-0082}
}
EUVIMED