Performance of ChatGPT on the Taiwan urology board examination: insights into current strengths and shortcomings
World Journal of Urology · 2024
DOI: 10.1007/s00345-024-04957-8 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift World Journal of Urology, 2024. Verfasst von Chung-You Tsai und 5 weiteren Autor:innen. Beteiligte Einrichtung: Divisions of Urology, Department of Surgery, Far Eastern Memorial Hospital, No.21, Sec. 2, Nanya S. Rd., Banciao Dist., New Taipei City, 220, Taiwan. Hauptthemen laut MEDLINE: Urology, Educational Measurement. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 25 Literaturangaben.
Abstract (Originalsprache)
PURPOSE: To compare ChatGPT-4 and ChatGPT-3.5's performance on Taiwan urology board examination (TUBE), focusing on answer accuracy, explanation consistency, and uncertainty management tactics to minimize score penalties from incorrect responses across 12 urology domains. METHODS: 450 multiple-choice questions from TUBE(2020-2022) were presented to two models. Three urologists assessed correctness and consistency of each response. Accuracy quantifies correct answers; consistency assesses logic and coherence in explanations out of total responses, alongside a penalty reduction experiment with prompt variations. Univariate logistic regression was applied for subgroup comparison. RESULTS: ChatGPT-4 showed strengths in urology, achieved an overall accuracy of 57.8%, with annual accuracies of 64.7% (2020), 58.0% (2021), and 50.7% (2022), significantly surpassing ChatGPT-3.5 (33.8%, OR = 2.68, 95% CI [2.05-3.52]). It could have passed the TUBE written exams if solely based on accuracy but failed in the final score due to penalties. ChatGPT-4 displayed a declining accuracy trend over time. Variability in accuracy across 12 urological domains was noted, with more frequently updated knowledge domains showing lower accuracy (53.2% vs. 62.2%, OR = 0.69, p = 0.05). A high consistency rate of 91.6% in explanations across all domains indicates reliable delivery of coherent and logical information. The simple prompt outperformed strategy-based prompts in accuracy (60% vs. 40%, p = 0.016), highlighting ChatGPT's limitations in its inability to accurately self-assess uncertainty and a tendency towards overconfidence, which may hinder medical decision-making. CONCLUSIONS: ChatGPT-4's high accuracy and consistent explanations in urology board examination demonstrate its potential in medical information processing. However, its limitations in self-assessment and overconfidence necessitate caution in its application, especially for inexperienced users. These insights call for ongoing advancements of urology-specific AI tools.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Divisions of Urology, Department of Surgery, Far Eastern Memorial Hospital, No.21, Sec. 2, Nanya S. Rd., Banciao Dist., New Taipei City, 220, Taiwan.
- Department of Electrical Engineering, Yuan Ze University, Taoyuan, Taiwan.
- Department of Biomedical Engineering, College of Medicine and College of Engineering, National Taiwan University, Taipei, Taiwan.
- Divisions of Urology, Department of Surgery, Far Eastern Memorial Hospital, No.21, Sec. 2, Nanya S. Rd., Banciao Dist., New Taipei City, 220, Taiwan. zack00639@gmail.com.
- Department of Biomedical Engineering, College of Medicine and College of Engineering, National Taiwan University, Taipei, Taiwan. zack00639@gmail.com.
Zitieren
APA
Tsai, C. Y., Hsieh, S. J., Huang, H. H., Deng, J. H., Huang, Y. Y., & Cheng, P. Y. (2024). Performance of ChatGPT on the Taiwan urology board examination: insights into current strengths and shortcomings. World Journal of Urology. https://doi.org/10.1007/s00345-024-04957-8
Vancouver
Tsai CY, Hsieh SJ, Huang HH, Deng JH, Huang YY, Cheng PY. Performance of ChatGPT on the Taiwan urology board examination: insights into current strengths and shortcomings. World Journal of Urology. 2024. doi:10.1007/s00345-024-04957-8
BibTeX
@article{Tsai2024,
author = {Tsai, Chung-You and Hsieh, Shang-Ju and Huang, Hung-Hsiang and Deng, Juinn-Horng and Huang, Yi-You and Cheng, Pai-Yu},
title = {Performance of ChatGPT on the Taiwan urology board examination: insights into current strengths and shortcomings},
journal = {World Journal of Urology},
year = {2024},
doi = {10.1007/s00345-024-04957-8}
}
EUVIMED