EUVIMEDEUVIMED entdecken
Delphi-Studie

Evaluation of large language models for clinical decision support in renal cell carcinoma: a multidisciplinary Delphi study

Sergio Fernández-Pello Montes, Ignacio Osman García, Rocio Barrabino Martín, María José Agustín Ferrández, Ferrán Algaba Arrea, Eliseo Carrasco Esteban, Bernardo Herrera Imbroda, María Carmen Mir Maresma, Carlos Nicolau, Marc Simó Perdigó, Guillermo de Velasco Oria de Rueda, Ángel Borque-Fernando

Journal of Comparative Effectiveness Research · 2026-11 · 15 (11s)

DOI: 10.57264/cer-2026-0082 ↗

Kurz gefasst

Delphi-Studie in der Zeitschrift Journal of Comparative Effectiveness Research, 2026-11. Verfasst von Sergio Fernández-Pello Montes und 11 weiteren Autor:innen. Beteiligte Einrichtung: Urology Department, Hospital Universitario de Cabueñes, Gijón, Spain. Hauptthemen laut MEDLINE: Large Language Models, Carcinoma, Renal Cell, Kidney Neoplasms, Decision Support Systems, Clinical. Die Arbeit nennt 27 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Aim

Large language models (LLMs) may facilitate the interpretation of clinical guideline information, but their performance in renal cell carcinoma (RCC) remains insufficiently characterized. We evaluated agreement between a multidisciplinary expert panel and the responses generated by Claude, Gemini and ChatGPT to RCC guideline-based questions. Materials & methods: A multidisciplinary tumor board developed 25 clinical questions based on the 2025 European Association of Urology RCC guidelines. Guideline figures and tables were converted into structured, plain-language textual summaries and supplied to the models. Each LLM generated one response per question using direct prompting. An 11-member multidisciplinary expert panel evaluated the responses in a two-round modified Delphi process using a 9-point scale and predefined consensus criteria.

Results

Consensus in agreement was reached for 24/25 Gemini responses (96%) and 23/25 responses (92%) for both Claude and ChatGPT. No response reached consensus in disagreement. The proportion of responses with a median score ≥7 was 25/25 (100%) for Gemini, 24/25 (96%) for Claude and 23/25 (92%) for ChatGPT. No statistically significant differences were detected among the models overall or in pairwise comparisons. Responses requiring further discussion or failing to reach consensus mainly involved clinically uncertain or individualized scenarios. Panelists identified occasional factual inaccuracies, omitted options and limited clinical nuance.

Conclusion

When provided with structured, guideline-derived material, selected LLMs generated responses that frequently aligned with multidisciplinary expert consensus. These findings support their potential for guideline interpretation but do not demonstrate independent clinical reasoning or real-world decision-making performance. Expert oversight remains essential, particularly in complex scenarios.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsCarcinoma, Renal CellKidney NeoplasmsDecision Support Systems, ClinicalHumansDelphi TechniquePractice Guidelines as TopicConsensus

Stichwörter der Autor:innen

Delphi techniqueartificial intelligenceclinical decision support systemsconsensusguideline adherencelarge language modelsrenal cell carcinoma

Autor:innen und Institutionen

  • Urology Department, Hospital Universitario de Cabueñes, Gijón, Spain.
  • Urology & Nephrology Department, Hospital Universitario Virgen del Rocío, Instituto Biomédico de Sevilla (Ibis), Universidad de Sevilla, Sevilla, Spain.
  • Urology Department, Hospital Universitario Torrecárdenas, Almeria, Spain.
  • Pharmacy Department, Hospital Universitario Miguel Servet, Zaragoza, Spain.
  • Pathology Department, Fundación Puigvert, Barcelona, Spain.
  • Radiotherapy Department, Hospital Central de la Defensa Gomez Ulla, Madrid, Spain.

Zitieren

APA

Montes, S. F., García, I. O., Martín, R. B., Agustín Ferrández, M. J., Arrea, F. A., Esteban, E. C., Imbroda, B. H., Mir Maresma, M. C., Nicolau, C., Perdigó, M. S., Velasco Oria de Rueda, G., & Borque-Fernando, Á. (2026). Evaluation of large language models for clinical decision support in renal cell carcinoma: a multidisciplinary Delphi study. Journal of Comparative Effectiveness Research, 15(11s). https://doi.org/10.57264/cer-2026-0082

Vancouver

Montes SF, García IO, Martín RB, Agustín Ferrández MJ, Arrea FA, Esteban EC, et al. Evaluation of large language models for clinical decision support in renal cell carcinoma: a multidisciplinary Delphi study. Journal of Comparative Effectiveness Research. 2026;15(11s). doi:10.57264/cer-2026-0082

BibTeX
@article{Montes2026,
  author = {Montes, Sergio Fernández-Pello and García, Ignacio Osman and Martín, Rocio Barrabino and Agustín Ferrández, María José and Arrea, Ferrán Algaba and Esteban, Eliseo Carrasco and Imbroda, Bernardo Herrera and Mir Maresma, María Carmen and Nicolau, Carlos and Perdigó, Marc Simó and Velasco Oria de Rueda, Guillermo de and Borque-Fernando, Ángel},
  title = {Evaluation of large language models for clinical decision support in renal cell carcinoma: a multidisciplinary Delphi study},
  journal = {Journal of Comparative Effectiveness Research},
  year = {2026},
  volume = {15},
  number = {11s},
  doi = {10.57264/cer-2026-0082}
}
Kostenlos starten →