Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study
Leistung von ChatGPT, Claude und AMBOSS im European Board of Urology In-Service Assessment und Übereinstimmung mit den EAU-Leitlinien 2025: vergleichende Studie
JMIR Formative Research · 02.09.2026 · 10 · e100148
DOI: 10.2196/100148 ↗Auf einen Blick
In 200 EBU-ähnlichen Multiple-Choice-Fragen erreichte ChatGPT die höchste Trefferquote. Auch die Übereinstimmung von Antwort und Begründung war bei ChatGPT am besten.
Fragestellung
Wie schneiden ChatGPT, Claude und AMBOSS bei EBU-ähnlichen Fragen zu Genauigkeit, Übereinstimmung und Leitlinienkonformität ab?
Methode
200 Single-Best-Answer-Fragen aus dem EBU In-Service Assessment Workbook 2021-2022 wurden in 3 KI-Modelle eingegeben; Bewertung durch 2 Urolog:innen.
Ergebnisse
Accuracy: ChatGPT 171/200 (85,5 %), Claude 159/200 (79,5 %), AMBOSS 159/200 (79,5 %; P=.14). Concordance: 190/200 (95 %), 176/200 (88 %), 152/200 (76 %; P0.61).
Schlussfolgerung der Autor:innen
KI-Tools können bei EBU-ähnlichen Prüfungen eine hohe Genauigkeit erreichen; Unterschiede in Denkqualität und Leitlinienadhärenz sind jedoch erkennbar. ChatGPT schnitt in allen Bereichen am besten ab.
Kernaussagen
- ChatGPT hatte die höchste Accuracy: 171/200 (85,5 %)
- Concordance war bei ChatGPT höher als bei Claude und AMBOSS
- Die Autor:innen sehen KI als mögliche Ergänzung in der urologischen Weiterbildung
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Vergleichsstudie in der Zeitschrift JMIR Formative Research, online erschienen am 02.09.2026. Verfasst von Mohamed Eldaneen und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Urology, Chelsea and Westminster Hospital NHS Foundation Trust, London, United Kingdom. Hauptthemen laut MEDLINE: Urology, Educational Measurement. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Recent advances in AI, particularly large language models, have generated growing interest in their application to medical education and examination preparation. However, the accuracy, reasoning quality, and adherence to clinical guidelines of these tools in postgraduate urology assessments remain unclear.
Objective
This study aimed to evaluate the performance of 3 AI tools, ChatGPT (GPT-4.0), Claude (version 4.5), and AMBOSS, on European Board of Urology (EBU)–style multiple-choice questions, with a particular focus on accuracy, insight, concordance, and adherence to European Association of Urology (EAU) guidelines.
Methods
A total of 200 single-best-answer questions from the EBU In-Service Assessment workbook (2021-2022) were input into each AI model. Models were prompted to select an answer and provide an explanation. Two urologists with post–Fellowship of the Royal College of Surgeons (FRCS) training independently assessed the outputs. Accuracy was defined as correct answer selection. Concordance was defined as the logical alignment between the answer and its explanation. Insight was evaluated across 3 domains—nonobvious deduction, discriminative reasoning, and clinical validity—and was graded as low, moderate, or high.
Results
ChatGPT demonstrated the highest accuracy (171/200, 85.5%), compared to Claude and AMBOSS (both 159/200, 79.5%; P=.14). Concordance was also significantly higher for ChatGPT (190/200, 95%) than for Claude (176/200, 88%) and AMBOSS (152/200, 76%; P0.61).
Conclusions
AI tools can achieve high accuracy on EBU-style assessments; however, differences in reasoning quality and guideline adherence are evident. ChatGPT demonstrated superior performance across all evaluated domains, supporting its role as a potential adjunct in postgraduate urology education.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Urology, Chelsea and Westminster Hospital NHS Foundation Trust, London, United Kingdom.
- East Kent Hospitals University NHS Foundation Trust, Kent, United Kingdom.
- Division of Surgery and Interventional Science, University College London, London, WC1E 6BT, United Kingdom.
- University College London Hospitals NHS Foundation Trust, London, England, United Kingdom.
ORCID-Profile: Mohamed Eldaneen, Shaza Hendy, Omar Ramadan, Panagiotis Nikolinakos, Asif Muneer, Hussain M Alnajjar, Karl H Pang
Zitieren
APA
Eldaneen, M., Hendy, S., Ramadan, O., Nikolinakos, P., Muneer, A., Alnajjar, H. M., & Pang, K. H. (2026). Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study. JMIR Formative Research, 10, e100148. https://doi.org/10.2196/100148
Vancouver
Eldaneen M, Hendy S, Ramadan O, Nikolinakos P, Muneer A, Alnajjar HM, et al. Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study. JMIR Formative Research. 2026;10:e100148. doi:10.2196/100148
BibTeX
@article{Eldaneen2026,
author = {Eldaneen, Mohamed and Hendy, Shaza and Ramadan, Omar and Nikolinakos, Panagiotis and Muneer, Asif and Alnajjar, Hussain M and Pang, Karl H},
title = {Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study},
journal = {JMIR Formative Research},
year = {2026},
volume = {10},
pages = {e100148},
doi = {10.2196/100148}
}
EUVIMED