EUVIMEDDescubrir EUVIMED
VergleichsstudieLeitlinieOpen access

Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study

Leistung von ChatGPT, Claude und AMBOSS im European Board of Urology In-Service Assessment und Übereinstimmung mit den EAU-Leitlinien 2025: vergleichende Studie

Mohamed Eldaneen iD, Shaza Hendy iD, Omar Ramadan iD, Panagiotis Nikolinakos iD, Asif Muneer iD, Hussain M Alnajjar iD, Karl H Pang iD

JMIR Formative Research · 2026-09-02 · 10 · e100148

DOI: 10.2196/100148 ↗

Auf einen Blick

In 200 EBU-ähnlichen Multiple-Choice-Fragen erreichte ChatGPT die höchste Trefferquote. Auch die Übereinstimmung von Antwort und Begründung war bei ChatGPT am besten.

Fragestellung

Wie schneiden ChatGPT, Claude und AMBOSS bei EBU-ähnlichen Fragen zu Genauigkeit, Übereinstimmung und Leitlinienkonformität ab?

Methode

200 Single-Best-Answer-Fragen aus dem EBU In-Service Assessment Workbook 2021-2022 wurden in 3 KI-Modelle eingegeben; Bewertung durch 2 Urolog:innen.

Ergebnisse

Accuracy: ChatGPT 171/200 (85,5 %), Claude 159/200 (79,5 %), AMBOSS 159/200 (79,5 %; P=.14). Concordance: 190/200 (95 %), 176/200 (88 %), 152/200 (76 %; P0.61).

Schlussfolgerung der Autor:innen

KI-Tools können bei EBU-ähnlichen Prüfungen eine hohe Genauigkeit erreichen; Unterschiede in Denkqualität und Leitlinienadhärenz sind jedoch erkennbar. ChatGPT schnitt in allen Bereichen am besten ab.

Kernaussagen

  • ChatGPT hatte die höchste Accuracy: 171/200 (85,5 %)
  • Concordance war bei ChatGPT höher als bei Claude und AMBOSS
  • Die Autor:innen sehen KI als mögliche Ergänzung in der urologischen Weiterbildung

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Vergleichsstudie in JMIR Formative Research, published online on 2026-09-02. Written by Mohamed Eldaneen and 6 co-authors. Institution: Department of Urology, Chelsea and Westminster Hospital NHS Foundation Trust, London, United Kingdom. Major MEDLINE topics: Urology, Educational Measurement. Cited 1 times so far (Crossref); 37 references. The full text is freely available in PubMed Central.

Auf einen BlickResumenSubjectsDatos bibliográficosCiteRelated studies

Abstract (original language)

Background

Recent advances in AI, particularly large language models, have generated growing interest in their application to medical education and examination preparation. However, the accuracy, reasoning quality, and adherence to clinical guidelines of these tools in postgraduate urology assessments remain unclear.

Objective

This study aimed to evaluate the performance of 3 AI tools, ChatGPT (GPT-4.0), Claude (version 4.5), and AMBOSS, on European Board of Urology (EBU)–style multiple-choice questions, with a particular focus on accuracy, insight, concordance, and adherence to European Association of Urology (EAU) guidelines.

Methods

A total of 200 single-best-answer questions from the EBU In-Service Assessment workbook (2021-2022) were input into each AI model. Models were prompted to select an answer and provide an explanation. Two urologists with post–Fellowship of the Royal College of Surgeons (FRCS) training independently assessed the outputs. Accuracy was defined as correct answer selection. Concordance was defined as the logical alignment between the answer and its explanation. Insight was evaluated across 3 domains—nonobvious deduction, discriminative reasoning, and clinical validity—and was graded as low, moderate, or high.

Results

ChatGPT demonstrated the highest accuracy (171/200, 85.5%), compared to Claude and AMBOSS (both 159/200, 79.5%; P=.14). Concordance was also significantly higher for ChatGPT (190/200, 95%) than for Claude (176/200, 88%) and AMBOSS (152/200, 76%; P0.61).

Conclusions

AI tools can achieve high accuracy on EBU-style assessments; however, differences in reasoning quality and guideline adherence are evident. ChatGPT demonstrated superior performance across all evaluated domains, supporting its role as a potential adjunct in postgraduate urology education.

Subjects

MeSH terms (MEDLINE)

UrologyEducational MeasurementHumansEuropeLarge Language ModelsGenerative Artificial Intelligence

Author keywords

AIAMBOSSChatGPTClaudeEAU guidelinesEBUEuropean Association of Urology guidelinesEuropean Board of Urologyartificial intelligence

Authors and institutions

  • Department of Urology, Chelsea and Westminster Hospital NHS Foundation Trust, London, United Kingdom.
  • East Kent Hospitals University NHS Foundation Trust, Kent, United Kingdom.
  • Division of Surgery and Interventional Science, University College London, London, WC1E 6BT, United Kingdom.
  • University College London Hospitals NHS Foundation Trust, London, England, United Kingdom.

ORCID profiles: Mohamed Eldaneen, Shaza Hendy, Omar Ramadan, Panagiotis Nikolinakos, Asif Muneer, Hussain M Alnajjar, Karl H Pang

Cite

APA

Eldaneen, M., Hendy, S., Ramadan, O., Nikolinakos, P., Muneer, A., Alnajjar, H. M., & Pang, K. H. (2026). Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study. JMIR Formative Research, 10, e100148. https://doi.org/10.2196/100148

Vancouver

Eldaneen M, Hendy S, Ramadan O, Nikolinakos P, Muneer A, Alnajjar HM, et al. Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study. JMIR Formative Research. 2026;10:e100148. doi:10.2196/100148

BibTeX
@article{Eldaneen2026,
  author = {Eldaneen, Mohamed and Hendy, Shaza and Ramadan, Omar and Nikolinakos, Panagiotis and Muneer, Asif and Alnajjar, Hussain M and Pang, Karl H},
  title = {Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study},
  journal = {JMIR Formative Research},
  year = {2026},
  volume = {10},
  pages = {e100148},
  doi = {10.2196/100148}
}
Start for free →