Evaluating the Efficacy of AI Chatbots as Tutors in Urology: A Comparative Analysis of Responses to the 2022 In-Service Assessment of the European Board of Urology
Urologia Internationalis · 2024
DOI: 10.1159/000537854 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Urologia Internationalis, 2024. Verfasst von Matthias May und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Urology, St. Elisabeth Hospital Straubing, Brothers of Mercy Hospital, Straubing, Germany. Hauptthemen laut MEDLINE: Urology. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Introduction
This study assessed the potential of large language models (LLMs) as educational tools by evaluating their accuracy in answering questions across urological subtopics.
Methods
Three LLMs (ChatGPT-3.5, ChatGPT-4, and Bing AI) were examined in two testing rounds, separated by 48 h, using 100 Multiple-Choice Questions (MCQs) from the 2022 European Board of Urology (EBU) In-Service Assessment (ISA), covering five different subtopics. The correct answer was defined as “formal accuracy” (FA) representing the designated single best answer (SBA) among four options. Alternative answers selected from LLMs, which may not necessarily be the SBA but are still deemed correct, were labeled as “extended accuracy” (EA). Their capacity to enhance the overall accuracy rate when combined with FA was examined.
Results
In two rounds of testing, the FA scores were achieved as follows: ChatGPT-3.5: 58% and 62%, ChatGPT-4: 63% and 77%, and BING AI: 81% and 73%. The incorporation of EA did not yield a significant enhancement in overall performance. The achieved gains for ChatGPT-3.5, ChatGPT-4, and BING AI were as a result 7% and 5%, 5% and 2%, and 3% and 1%, respectively (p > 0.3). Within urological subtopics, LLMs showcased best performance in Pediatrics/Congenital and comparatively less effectiveness in Functional/BPS/Incontinence.
Conclusion
LLMs exhibit suboptimal urology knowledge and unsatisfactory proficiency for educational purposes. The overall accuracy did not significantly improve when combining EA to FA. The error rates remained high ranging from 16 to 35%. Proficiency levels vary substantially across subtopics. Further development of medicine-specific LLMs is required before integration into urological training programs.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Urology, St. Elisabeth Hospital Straubing, Brothers of Mercy Hospital, Straubing, Germany.
- Department of Urology, Caritas St. Josef Medical Centre, University of Regensburg, Regensburg, Germany.
- Department of Urology and Andrology, Klinik Donaustadt, Vienna, Austria.
- Department of Urology, University of Munich, LMU, Munich, Germany.
- Johnson and Johnson Innovative Medicine, Research and Development, Spring House, Pennsylvania, USA.
- European Board of Urology, Arnhem, The Netherlands.
Zitieren
APA
May, M., Körner-Riffard, K., Kollitsch, L., Burger, M., Brookman-May, S. D., Rauchenwald, M., Marszalek, M., & Eredics, K. (2024). Evaluating the Efficacy of AI Chatbots as Tutors in Urology: A Comparative Analysis of Responses to the 2022 In-Service Assessment of the European Board of Urology. Urologia Internationalis. https://doi.org/10.1159/000537854
Vancouver
May M, Körner-Riffard K, Kollitsch L, Burger M, Brookman-May SD, Rauchenwald M, et al. Evaluating the Efficacy of AI Chatbots as Tutors in Urology: A Comparative Analysis of Responses to the 2022 In-Service Assessment of the European Board of Urology. Urologia Internationalis. 2024. doi:10.1159/000537854
BibTeX
@article{May2024,
author = {May, Matthias and Körner-Riffard, Katharina and Kollitsch, Lisa and Burger, Maximilian and Brookman-May, Sabine D and Rauchenwald, Michael and Marszalek, Martin and Eredics, Klaus},
title = {Evaluating the Efficacy of AI Chatbots as Tutors in Urology: A Comparative Analysis of Responses to the 2022 In-Service Assessment of the European Board of Urology},
journal = {Urologia Internationalis},
year = {2024},
doi = {10.1159/000537854}
}
EUVIMED