Accuracy, Completeness, and Clarity of an AI-Based Chatbot for the EAU Neuro-Urology Guidelines
European Urology Focus · 2026-08
DOI: 10.1016/j.euf.2026.06.025 ↗Kurz gefasst
Leitlinie in der Zeitschrift European Urology Focus, 2026-08. Verfasst von Sabrina De Cillis und 27 weiteren Autor:innen. Beteiligte Einrichtung: Division of Urology, San Luigi Gonzaga Hospital, Department of Oncology, University of Turin, Orbassano, Italy. Die Arbeit nennt 20 Literaturangaben.
Abstract (Originalsprache)
This study aimed to externally validate the performance of the European Association of Urology (EAU) Guidelines Bot in neuro-urology by assessing the accuracy, completeness, and clarity of chatbot-generated answers to guideline-based questions and to compare its performance with that of a general-purpose large language model (ChatGPT 5.5). A cross-sectional validation study was conducted using 47 questions derived from the EAU Neuro-Urology Guidelines. Each question was linked to a specific recommendation and classified by recommendation strength (strong vs weak). Questions were independently submitted to both the EAU Guidelines Bot and ChatGPT 5.5 without additional prompting. Two expert urologists independently evaluated each response for accuracy, completeness, and clarity using a five-point Likert scale; discrepancies were resolved by a third reviewer. Overall, 45 questions (95.7%) were linked to strong recommendations and two (4.3%) to weak recommendations. The EAU Guidelines Bot and ChatGPT 5.5 achieved identical mean accuracy scores (4.96 ± 0.20), with all responses rated as highly accurate (Likert 4-5). ChatGPT 5.5 indicated significantly higher completeness scores than did the EAU Guidelines Bot (4.74 ± 0.44 vs 4.57 ± 0.54; p = 0.011), whereas clarity scores were not significantly different (4.83 ± 0.38 vs 4.77 ± 0.43; p = 0.083). High-quality completeness was observed in 46/47 EAU Guidelines Bot responses (97.9%) and 47/47 ChatGPT responses (100%). Score discrepancies between systems were identified in ten of 47 questions (21.3%) and were limited to completeness and clarity domains. Performance remained uniformly high across recommendation grades, with no meaningful differences observed. The EAU Guidelines Bot showed excellent accuracy, completeness, and clarity when applied to neuro-urology guideline-based questions. Its performance was comparable to that of ChatGPT 5.5, with both systems providing highly accurate guideline-concordant responses. Although ChatGPT 5.5 generated more comprehensive answers, the EAU Guidelines Bot maintained closer adherence to the original guideline recommendations. Although not a substitute for clinical judgment, the tool appears to be a reliable adjunct for rapid access to evidence-based neuro-urological guidance.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Division of Urology, San Luigi Gonzaga Hospital, Department of Oncology, University of Turin, Orbassano, Italy.
- Department of Urology, Sapienza University of Rome, Rome, Italy. Electronic address: rlombardo@me.com.
- Department of Oncology, University of Turin, Orbassano, Italy; Division of Urology, Department of Surgery, Candiolo Cancer Institute FPO-IRCCS, Candiolo, Italy.
- Department of Urology, Sapienza University of Rome, Rome, Italy.
- Department of Urology, University of Tor Vergata, Rome, Italy.
- Department of Urology, Gemelli University, Rome, Italy.
Zitieren
APA
De Cillis, S., Lombardo, R., Amparore, D., Romagnoli, M., Santioni, M., Sequi, M. B., Piramide, F., Piana, A., Quara', A., Manfredi, M., Rosato, E., Ragonese, M., Santarelli, V., Spatafora, P., Gallo, G., Pagliarulo, V., Tema, G., Franco, A., Pastore, A. L., …, & Next-Gen Research Group (2026). Accuracy, Completeness, and Clarity of an AI-Based Chatbot for the EAU Neuro-Urology Guidelines. European Urology Focus. https://doi.org/10.1016/j.euf.2026.06.025
Vancouver
De Cillis S, Lombardo R, Amparore D, Romagnoli M, Santioni M, Sequi MB, et al. Accuracy, Completeness, and Clarity of an AI-Based Chatbot for the EAU Neuro-Urology Guidelines. European Urology Focus. 2026. doi:10.1016/j.euf.2026.06.025
BibTeX
@article{DeCillis2026,
author = {De Cillis, Sabrina and Lombardo, Riccardo and Amparore, Daniele and Romagnoli, Matteo and Santioni, Marta and Sequi, Manfredi Bruno and Piramide, Federico and Piana, Alberto and Quara', Alberto and Manfredi, Matteo and Rosato, Eleonora and Ragonese, Mauro and Santarelli, Valerio and Spatafora, Pietro and Gallo, Giacomo and Pagliarulo, Vincenzo and Tema, Giorgia and Franco, Antonio and Pastore, Antonio Luigi and Rocco, Bernardo and Gacci, Mauro and Fiori, Cristian and Finazzi, Enrico and Sciarra, Alessandro and Tubaro, Andrea and Porpiglia, Francesco and De Nunzio, Cosimo and Next-Gen Research Group, Next-Gen Research Group},
title = {Accuracy, Completeness, and Clarity of an AI-Based Chatbot for the EAU Neuro-Urology Guidelines},
journal = {European Urology Focus},
year = {2026},
doi = {10.1016/j.euf.2026.06.025}
}
EUVIMED