Evaluation of large language models for guideline-based clinical questions in muscle-invasive and non-muscle-invasive bladder cancer and upper tract urothelial carcinoma: a multidisciplinary Delphi study
Journal of Comparative Effectiveness Research · 2026-11 · 15 (11s)
DOI: 10.57264/cer-2026-0079 ↗Kurz gefasst
Leitlinie in der Zeitschrift Journal of Comparative Effectiveness Research, 2026-11. Verfasst von Félix Guerrero-Ramos und 26 weiteren Autor:innen. Beteiligte Einrichtung: Urology Department, Hospital Universitario 12 de Octubre, Madrid, Spain. Hauptthemen laut MEDLINE: Large Language Models, Urinary Bladder Neoplasms, Practice Guidelines as Topic, Carcinoma, Transitional Cell. Die Arbeit nennt 22 Literaturangaben.
Abstract (Originalsprache)
Aim
To assess expert panel agreement with responses from three large language models, Claude, Gemini and ChatGPT, to questions based on guidelines for non-muscle-invasive bladder cancer (NMIBC), muscle-invasive bladder cancer (MIBC) and upper tract urothelial carcinoma (UTUC). Materials & methods: The 2025 European Association of Urology (EAU) guidelines were provided to Claude 3.5 Haiku, Gemini 2.5 Flash and GPT-4.1-mini. Figures and tables were converted into plain language text. Each model answered 25 questions per entity. Panels of 7–10 experts rated responses on a 9-point scale using a modified Delphi process with two rounds. Outcomes were consensus in agreement and median expert panel score ≥7.
Results
For Claude, Gemini and ChatGPT, respectively, consensus in agreement was achieved for 88%, 100% and 96% of NMIBC responses (p = 0.097); 92%, 100% and 96% of MIBC responses (p = 0.223); and 56%, 100% and 96% of UTUC responses (p < 0.001). For UTUC, significantly fewer Claude responses achieved consensus in agreement than Gemini or ChatGPT responses. Across all questions, consensus in agreement was achieved for 59/75 Claude responses (78.7%), 75/75 Gemini responses (100%), and 72/75 ChatGPT responses (96%) (p < 0.001). Median scores ≥7 were achieved for 88%, 100% and 100% of NMIBC responses; 96%, 100% and 100% of MIBC responses; and 68%, 100% and 96% of UTUC responses. In UTUC, Claude had significantly fewer such responses than Gemini, but the difference from ChatGPT was not statistically significant. Overall, median scores ≥7 were achieved for 63/75 Claude responses (84%), 75/75 Gemini responses (100%) and 74/75 ChatGPT responses (98.7%) (p < 0.001).
Conclusion
Under the study conditions, large language model responses frequently achieved expert panel agreement, particularly for Gemini and ChatGPT, supporting the potential of these models for guideline interpretation. Further validation is required to establish clinical utility and safety.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Urology Department, Hospital Universitario 12 de Octubre, Madrid, Spain.
- Urology Department, Hospital Universitario Central de Asturias, Universidad de Oviedo, Oviedo, Spain.
- Urology Department, Hospital General Universitario Morales Meseguer, Murcia, Spain.
- Department of Morphological Sciences, Anatomical Pathology Unit, Universidad de Córdoba, Córdoba, Spain.
- Urology Department, Hospital Universitari de Bellvitge, L'Hospitalet de Llobregat, Barcelona, Spain.
- Urology Department, Hospital Universitario de Getafe, Getafe, Madrid, Spain.
Zitieren
APA
Guerrero-Ramos, F., Fernández Gómez, J. M., Aparicio, T. F., López-Beltrán, A., Rueda, Ó. B., González, L. L., Vázquez, J. P., Carrión, A., Iriarte, A. L., Subiela Henríquez, J. D., Couñago, F., Benavides, J. M., Gómez, P. P., Juan Fita, M. J., Vilaseca, R. M., Vilaseca, A., Sureda, L. L., de Lorca, A. P., Domínguez Escrig, J. L., …, & Borque-Fernando, Á. (2026). Evaluation of large language models for guideline-based clinical questions in muscle-invasive and non-muscle-invasive bladder cancer and upper tract urothelial carcinoma: a multidisciplinary Delphi study. Journal of Comparative Effectiveness Research, 15(11s). https://doi.org/10.57264/cer-2026-0079
Vancouver
Guerrero-Ramos F, Fernández Gómez JM, Aparicio TF, López-Beltrán A, Rueda ÓB, González LL, et al. Evaluation of large language models for guideline-based clinical questions in muscle-invasive and non-muscle-invasive bladder cancer and upper tract urothelial carcinoma: a multidisciplinary Delphi study. Journal of Comparative Effectiveness Research. 2026;15(11s). doi:10.57264/cer-2026-0079
BibTeX
@article{GuerreroRamos2026,
author = {Guerrero-Ramos, Félix and Fernández Gómez, Jesús María and Aparicio, Tomás Fernández and López-Beltrán, Antonio and Rueda, Óscar Buisán and González, Luis Llanes and Vázquez, Javier Puente and Carrión, Alberto and Iriarte, Ana Loizaga and Subiela Henríquez, José Daniel and Couñago, Felipe and Benavides, Javier Martínez and Gómez, Paula Pelechano and Juan Fita, María José and Vilaseca, Richard Mast and Vilaseca, Antoni and Sureda, Luis Ladaria and de Lorca, Alberto Pérez-Lanzac and Domínguez Escrig, José Luis and Ortega, Julián Sanz and Fernández, Elena Sevillano and Plata Bello, Ana Cristina and Auba, Felipe Villacampa and Villa, Marta Trassierra and Vargas, Eloy Vivas and Puigsureda, Isabel Trias and Borque-Fernando, Ángel},
title = {Evaluation of large language models for guideline-based clinical questions in muscle-invasive and non-muscle-invasive bladder cancer and upper tract urothelial carcinoma: a multidisciplinary Delphi study},
journal = {Journal of Comparative Effectiveness Research},
year = {2026},
volume = {15},
number = {11s},
doi = {10.57264/cer-2026-0079}
}
EUVIMED