Evaluation of large language models for clinical decision support in penile and testicular cancer: a multidisciplinary Delphi study
Journal of Comparative Effectiveness Research · 2026-11 · 15 (11s)
DOI: 10.57264/cer-2026-0080 ↗Kurz gefasst
Delphi-Studie in der Zeitschrift Journal of Comparative Effectiveness Research, 2026-11. Verfasst von José Rubio-Briones und 18 weiteren Autor:innen. Beteiligte Einrichtung: Urology Department, Hospital VITHAS 9 de Octubre, Valencia, Spain. Hauptthemen laut MEDLINE: Large Language Models, Testicular Neoplasms, Penile Neoplasms, Decision Support Systems, Clinical. Die Arbeit nennt 19 Literaturangaben.
Abstract (Originalsprache)
Aim
Large language models (LLMs) may help interpret clinical guideline information, but their performance in penile and testicular cancer remains poorly defined. We evaluated agreement between multidisciplinary expert panels and responses generated by Claude 3.5 Haiku, Gemini 2.5 Flash and GPT-4.1-mini to guideline-based questions on these cancers. Materials & methods: Specialized tumor boards developed 25 questions for each cancer based on the 2025 European Association of Urology guidelines. The original guideline text, with figures and tables converted into structured, plain-language textual descriptions, was provided to each LLM. Separate panels of nine experts evaluated the responses using a two-round modified Delphi process with predefined consensus criteria.
Results
For penile cancer, consensus in agreement was reached on 21/25 responses (84%) for Claude, 24/25 (96%) for Gemini and 23/25 (92%) for ChatGPT, with no statistically significant differences in overall or pairwise comparisons. For testicular cancer, consensus in agreement was reached on 24/25 items (96%) for each model. One Claude response and one ChatGPT response reached consensus in disagreement. No significant inter-model differences were observed. In both cancers, most responses not reaching consensus or requiring a second round related to prognosis or staging. Across both cancers combined, consensus in agreement was 90% for Claude, 96% for Gemini and 94% for ChatGPT, with no significant inter-model differences. Median scores ≥7 occurred in 94–98% of responses for all LLMs.
Conclusion
Selected lightweight variants from widely used LLM families showed substantial agreement with expert consensus when provided with guideline content, although findings reflect performance in a controlled, guideline-fed setting and require specialist validation before clinical application.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Urology Department, Hospital VITHAS 9 de Octubre, Valencia, Spain.
- Urology Department, Hospital Clínic Barcelona, Barcelona, Spain.
- Urology Department, Hospital Universitario Central de Asturias, Oviedo, Spain.
- Oncology Department, Complejo Hospitalario Universitario de Santiago de Compostela, Santiago de Compostela, Spain.
- Urology Department, Hospital General Universitario Morales Meseguer, Murcia, Spain.
- Urology Department, Complejo Hospitalario Universitario de Badajoz, Badajoz, Spain.
Zitieren
APA
Rubio-Briones, J., Reyes, L. I., Rodríguez, J. G., Herranz, U. A., Sánchez, A. R., Cabo González, J. A., Sánchez, A. S., Esteban, M. D., de Liaño Lista, A. G., Gutiérrez, P. G., Barona, R. L., Cardesa, O. G., Gómez-Aparicio, M. A., Muñoz Iglesias, J. L., Rodríguez-Fraile, M., Martínez, A. O., Gaya, J. M., Rodríguez-Bermejo, P. R., & Borque-Fernando, Á. (2026). Evaluation of large language models for clinical decision support in penile and testicular cancer: a multidisciplinary Delphi study. Journal of Comparative Effectiveness Research, 15(11s). https://doi.org/10.57264/cer-2026-0080
Vancouver
Rubio-Briones J, Reyes LI, Rodríguez JG, Herranz UA, Sánchez AR, Cabo González JA, et al. Evaluation of large language models for clinical decision support in penile and testicular cancer: a multidisciplinary Delphi study. Journal of Comparative Effectiveness Research. 2026;15(11s). doi:10.57264/cer-2026-0080
BibTeX
@article{RubioBriones2026,
author = {Rubio-Briones, José and Reyes, Laura Izquierdo and Rodríguez, Jorge García and Herranz, Urbano Anido and Sánchez, Antonio Rosino and Cabo González, Juan Alonso and Sánchez, Antonio Salinas and Esteban, Mario Domínguez and de Liaño Lista, Alfonso Gómez and Gutiérrez, Pilar González-Peramato and Barona, Rafael Luque and Cardesa, Oscar Gorria and Gómez-Aparicio, María Antonia and Muñoz Iglesias, José Lorenzo and Rodríguez-Fraile, Macarena and Martínez, Abrahams Ocanto and Gaya, Josep M and Rodríguez-Bermejo, Patricia Ramírez and Borque-Fernando, Ángel},
title = {Evaluation of large language models for clinical decision support in penile and testicular cancer: a multidisciplinary Delphi study},
journal = {Journal of Comparative Effectiveness Research},
year = {2026},
volume = {15},
number = {11s},
doi = {10.57264/cer-2026-0080}
}
EUVIMED