EUVIMEDEUVIMED entdecken
VergleichsstudieLeitlinieOpen Access

Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study

Leistung von ChatGPT, Claude und AMBOSS im European Board of Urology In-Service Assessment und Übereinstimmung mit den EAU-Leitlinien 2025: vergleichende Studie

Mohamed Eldaneen iD, Shaza Hendy iD, Omar Ramadan iD, Panagiotis Nikolinakos iD, Asif Muneer iD, Hussain M Alnajjar iD, Karl H Pang iD

JMIR Formative Research · 02.09.2026 · 10 · e100148

DOI: 10.2196/100148 ↗

Auf einen Blick

In 200 EBU-ähnlichen Multiple-Choice-Fragen erreichte ChatGPT die höchste Trefferquote. Auch die Übereinstimmung von Antwort und Begründung war bei ChatGPT am besten.

Fragestellung

Wie schneiden ChatGPT, Claude und AMBOSS bei EBU-ähnlichen Fragen zu Genauigkeit, Übereinstimmung und Leitlinienkonformität ab?

Methode

200 Single-Best-Answer-Fragen aus dem EBU In-Service Assessment Workbook 2021-2022 wurden in 3 KI-Modelle eingegeben; Bewertung durch 2 Urolog:innen.

Ergebnisse

Accuracy: ChatGPT 171/200 (85,5 %), Claude 159/200 (79,5 %), AMBOSS 159/200 (79,5 %; P=.14). Concordance: 190/200 (95 %), 176/200 (88 %), 152/200 (76 %; P0.61).

Schlussfolgerung der Autor:innen

KI-Tools können bei EBU-ähnlichen Prüfungen eine hohe Genauigkeit erreichen; Unterschiede in Denkqualität und Leitlinienadhärenz sind jedoch erkennbar. ChatGPT schnitt in allen Bereichen am besten ab.

Kernaussagen

  • ChatGPT hatte die höchste Accuracy: 171/200 (85,5 %)
  • Concordance war bei ChatGPT höher als bei Claude und AMBOSS
  • Die Autor:innen sehen KI als mögliche Ergänzung in der urologischen Weiterbildung

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

Kurz gefasst

Vergleichsstudie in der Zeitschrift JMIR Formative Research, online erschienen am 02.09.2026. Verfasst von Mohamed Eldaneen und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Urology, Chelsea and Westminster Hospital NHS Foundation Trust, London, United Kingdom. Hauptthemen laut MEDLINE: Urology, Educational Measurement. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

Auf einen BlickAbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Recent advances in AI, particularly large language models, have generated growing interest in their application to medical education and examination preparation. However, the accuracy, reasoning quality, and adherence to clinical guidelines of these tools in postgraduate urology assessments remain unclear.

Objective

This study aimed to evaluate the performance of 3 AI tools, ChatGPT (GPT-4.0), Claude (version 4.5), and AMBOSS, on European Board of Urology (EBU)–style multiple-choice questions, with a particular focus on accuracy, insight, concordance, and adherence to European Association of Urology (EAU) guidelines.

Methods

A total of 200 single-best-answer questions from the EBU In-Service Assessment workbook (2021-2022) were input into each AI model. Models were prompted to select an answer and provide an explanation. Two urologists with post–Fellowship of the Royal College of Surgeons (FRCS) training independently assessed the outputs. Accuracy was defined as correct answer selection. Concordance was defined as the logical alignment between the answer and its explanation. Insight was evaluated across 3 domains—nonobvious deduction, discriminative reasoning, and clinical validity—and was graded as low, moderate, or high.

Results

ChatGPT demonstrated the highest accuracy (171/200, 85.5%), compared to Claude and AMBOSS (both 159/200, 79.5%; P=.14). Concordance was also significantly higher for ChatGPT (190/200, 95%) than for Claude (176/200, 88%) and AMBOSS (152/200, 76%; P0.61).

Conclusions

AI tools can achieve high accuracy on EBU-style assessments; however, differences in reasoning quality and guideline adherence are evident. ChatGPT demonstrated superior performance across all evaluated domains, supporting its role as a potential adjunct in postgraduate urology education.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

UrologyEducational MeasurementHumansEuropeLarge Language ModelsGenerative Artificial Intelligence

Stichwörter der Autor:innen

AIAMBOSSChatGPTClaudeEAU guidelinesEBUEuropean Association of Urology guidelinesEuropean Board of Urologyartificial intelligence

Autor:innen und Institutionen

  • Department of Urology, Chelsea and Westminster Hospital NHS Foundation Trust, London, United Kingdom.
  • East Kent Hospitals University NHS Foundation Trust, Kent, United Kingdom.
  • Division of Surgery and Interventional Science, University College London, London, WC1E 6BT, United Kingdom.
  • University College London Hospitals NHS Foundation Trust, London, England, United Kingdom.

ORCID-Profile: Mohamed Eldaneen, Shaza Hendy, Omar Ramadan, Panagiotis Nikolinakos, Asif Muneer, Hussain M Alnajjar, Karl H Pang

Zitieren

APA

Eldaneen, M., Hendy, S., Ramadan, O., Nikolinakos, P., Muneer, A., Alnajjar, H. M., & Pang, K. H. (2026). Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study. JMIR Formative Research, 10, e100148. https://doi.org/10.2196/100148

Vancouver

Eldaneen M, Hendy S, Ramadan O, Nikolinakos P, Muneer A, Alnajjar HM, et al. Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study. JMIR Formative Research. 2026;10:e100148. doi:10.2196/100148

BibTeX
@article{Eldaneen2026,
  author = {Eldaneen, Mohamed and Hendy, Shaza and Ramadan, Omar and Nikolinakos, Panagiotis and Muneer, Asif and Alnajjar, Hussain M and Pang, Karl H},
  title = {Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study},
  journal = {JMIR Formative Research},
  year = {2026},
  volume = {10},
  pages = {e100148},
  doi = {10.2196/100148}
}
Kostenlos starten →