EUVIMEDEUVIMED entdecken
LeitlinieDelphi-Studie

Evaluation of large language models for guideline-based clinical questions in muscle-invasive and non-muscle-invasive bladder cancer and upper tract urothelial carcinoma: a multidisciplinary Delphi study

Félix Guerrero-Ramos, Jesús María Fernández Gómez, Tomás Fernández Aparicio, Antonio López-Beltrán, Óscar Buisán Rueda, Luis Llanes González, Javier Puente Vázquez, Alberto Carrión, Ana Loizaga Iriarte, José Daniel Subiela Henríquez, Felipe Couñago, Javier Martínez Benavides und 15 weitere

Journal of Comparative Effectiveness Research · 2026-11 · 15 (11s)

DOI: 10.57264/cer-2026-0079 ↗

Kurz gefasst

Leitlinie in der Zeitschrift Journal of Comparative Effectiveness Research, 2026-11. Verfasst von Félix Guerrero-Ramos und 26 weiteren Autor:innen. Beteiligte Einrichtung: Urology Department, Hospital Universitario 12 de Octubre, Madrid, Spain. Hauptthemen laut MEDLINE: Large Language Models, Urinary Bladder Neoplasms, Practice Guidelines as Topic, Carcinoma, Transitional Cell. Die Arbeit nennt 22 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Aim

To assess expert panel agreement with responses from three large language models, Claude, Gemini and ChatGPT, to questions based on guidelines for non-muscle-invasive bladder cancer (NMIBC), muscle-invasive bladder cancer (MIBC) and upper tract urothelial carcinoma (UTUC). Materials & methods: The 2025 European Association of Urology (EAU) guidelines were provided to Claude 3.5 Haiku, Gemini 2.5 Flash and GPT-4.1-mini. Figures and tables were converted into plain language text. Each model answered 25 questions per entity. Panels of 7–10 experts rated responses on a 9-point scale using a modified Delphi process with two rounds. Outcomes were consensus in agreement and median expert panel score ≥7.

Results

For Claude, Gemini and ChatGPT, respectively, consensus in agreement was achieved for 88%, 100% and 96% of NMIBC responses (p = 0.097); 92%, 100% and 96% of MIBC responses (p = 0.223); and 56%, 100% and 96% of UTUC responses (p < 0.001). For UTUC, significantly fewer Claude responses achieved consensus in agreement than Gemini or ChatGPT responses. Across all questions, consensus in agreement was achieved for 59/75 Claude responses (78.7%), 75/75 Gemini responses (100%), and 72/75 ChatGPT responses (96%) (p < 0.001). Median scores ≥7 were achieved for 88%, 100% and 100% of NMIBC responses; 96%, 100% and 100% of MIBC responses; and 68%, 100% and 96% of UTUC responses. In UTUC, Claude had significantly fewer such responses than Gemini, but the difference from ChatGPT was not statistically significant. Overall, median scores ≥7 were achieved for 63/75 Claude responses (84%), 75/75 Gemini responses (100%) and 74/75 ChatGPT responses (98.7%) (p < 0.001).

Conclusion

Under the study conditions, large language model responses frequently achieved expert panel agreement, particularly for Gemini and ChatGPT, supporting the potential of these models for guideline interpretation. Further validation is required to establish clinical utility and safety.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsUrinary Bladder NeoplasmsPractice Guidelines as TopicCarcinoma, Transitional CellHumansDelphi TechniqueNon-Muscle Invasive Bladder NeoplasmsConsensusFemaleMale

Stichwörter der Autor:innen

artificial intelligenceconsensuslarge language modelsupper urinary tract tumorurinary bladder neoplasmsurothelial cancer

Autor:innen und Institutionen

  • Urology Department, Hospital Universitario 12 de Octubre, Madrid, Spain.
  • Urology Department, Hospital Universitario Central de Asturias, Universidad de Oviedo, Oviedo, Spain.
  • Urology Department, Hospital General Universitario Morales Meseguer, Murcia, Spain.
  • Department of Morphological Sciences, Anatomical Pathology Unit, Universidad de Córdoba, Córdoba, Spain.
  • Urology Department, Hospital Universitari de Bellvitge, L'Hospitalet de Llobregat, Barcelona, Spain.
  • Urology Department, Hospital Universitario de Getafe, Getafe, Madrid, Spain.

Zitieren

APA

Guerrero-Ramos, F., Fernández Gómez, J. M., Aparicio, T. F., López-Beltrán, A., Rueda, Ó. B., González, L. L., Vázquez, J. P., Carrión, A., Iriarte, A. L., Subiela Henríquez, J. D., Couñago, F., Benavides, J. M., Gómez, P. P., Juan Fita, M. J., Vilaseca, R. M., Vilaseca, A., Sureda, L. L., de Lorca, A. P., Domínguez Escrig, J. L., …, & Borque-Fernando, Á. (2026). Evaluation of large language models for guideline-based clinical questions in muscle-invasive and non-muscle-invasive bladder cancer and upper tract urothelial carcinoma: a multidisciplinary Delphi study. Journal of Comparative Effectiveness Research, 15(11s). https://doi.org/10.57264/cer-2026-0079

Vancouver

Guerrero-Ramos F, Fernández Gómez JM, Aparicio TF, López-Beltrán A, Rueda ÓB, González LL, et al. Evaluation of large language models for guideline-based clinical questions in muscle-invasive and non-muscle-invasive bladder cancer and upper tract urothelial carcinoma: a multidisciplinary Delphi study. Journal of Comparative Effectiveness Research. 2026;15(11s). doi:10.57264/cer-2026-0079

BibTeX
@article{GuerreroRamos2026,
  author = {Guerrero-Ramos, Félix and Fernández Gómez, Jesús María and Aparicio, Tomás Fernández and López-Beltrán, Antonio and Rueda, Óscar Buisán and González, Luis Llanes and Vázquez, Javier Puente and Carrión, Alberto and Iriarte, Ana Loizaga and Subiela Henríquez, José Daniel and Couñago, Felipe and Benavides, Javier Martínez and Gómez, Paula Pelechano and Juan Fita, María José and Vilaseca, Richard Mast and Vilaseca, Antoni and Sureda, Luis Ladaria and de Lorca, Alberto Pérez-Lanzac and Domínguez Escrig, José Luis and Ortega, Julián Sanz and Fernández, Elena Sevillano and Plata Bello, Ana Cristina and Auba, Felipe Villacampa and Villa, Marta Trassierra and Vargas, Eloy Vivas and Puigsureda, Isabel Trias and Borque-Fernando, Ángel},
  title = {Evaluation of large language models for guideline-based clinical questions in muscle-invasive and non-muscle-invasive bladder cancer and upper tract urothelial carcinoma: a multidisciplinary Delphi study},
  journal = {Journal of Comparative Effectiveness Research},
  year = {2026},
  volume = {15},
  number = {11s},
  doi = {10.57264/cer-2026-0079}
}
Kostenlos starten →