Evaluating the Performance of ChatGPT in Urology: A Comparative Study of Knowledge Interpretation and Patient Guidance
Journal of Endourology · 2024-08 · 38 (8) · 799-808
DOI: 10.1089/end.2023.0413 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of Endourology, online erschienen am 01.08.2024. Verfasst von Bahadır Şahin und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Urology, Marmara University, School of Medicine, Marmara University, Istanbul, Turkey. Hauptthemen laut MEDLINE: Urology. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 13 Literaturangaben.
Abstract (Originalsprache)
Background/Aim: To evaluate the performance of Chat Generative Pre-trained Transformer (ChatGPT), a large language model trained by Open artificial intelligence. Materials and Methods: This study has three main steps to evaluate the effectiveness of ChatGPT in the urologic field. The first step involved 35 questions from our institution's experts, who have at least 10 years of experience in their fields. The responses of ChatGPT versions were qualitatively compared with the responses of urology residents to the same questions. The second step assesses the reliability of ChatGPT versions in answering current debate topics. The third step was to assess the reliability of ChatGPT versions in providing medical recommendations and directives to patients' commonly asked questions during the outpatient and inpatient clinic.
Results
In the first step, version 4 provided correct answers to 25 questions out of 35 while version 3.5 provided only 19 (71.4% vs 54%). It was observed that residents in their last year of education in our clinic also provided a mean of 25 correct answers, and 4th year residents provided a mean of 19.3 correct responses. The second step involved evaluating the response of both versions to debate situations in urology, and it was found that both versions provided variable and inappropriate results. In the last step, both versions had a similar success rate in providing recommendations and guidance to patients based on expert ratings.
Conclusion
The difference between the two versions of the 35 questions in the first step of the study was thought to be due to the improvement of ChatGPT's literature and data synthesis abilities. It may be a logical approach to use ChatGPT versions to inform the nonhealth care providers' questions with quick and safe answers but should not be used to as a diagnostic tool or make a choice among different treatment modalities.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Urology, Marmara University, School of Medicine, Marmara University, Istanbul, Turkey.
ORCID-Profile: Yunus Emre Genç, Yılören Tanıdır
Zitieren
APA
Şahin, B., Emre Genç, Y., Doğan, K., Emre Şener, T., Şekerci, Ç. A., Tanıdır, Y., Yücel, S., Tarcan, T., & Çam, H. K. (2024). Evaluating the Performance of ChatGPT in Urology: A Comparative Study of Knowledge Interpretation and Patient Guidance. Journal of Endourology, 38(8), 799-808. https://doi.org/10.1089/end.2023.0413
Vancouver
Şahin B, Emre Genç Y, Doğan K, Emre Şener T, Şekerci ÇA, Tanıdır Y, et al. Evaluating the Performance of ChatGPT in Urology: A Comparative Study of Knowledge Interpretation and Patient Guidance. Journal of Endourology. 2024;38(8):799-808. doi:10.1089/end.2023.0413
BibTeX
@article{ahin2024,
author = {Şahin, Bahadır and Emre Genç, Yunus and Doğan, Kader and Emre Şener, Tarık and Şekerci, Çağrı Akın and Tanıdır, Yılören and Yücel, Selçuk and Tarcan, Tufan and Çam, Haydar Kamil},
title = {Evaluating the Performance of ChatGPT in Urology: A Comparative Study of Knowledge Interpretation and Patient Guidance},
journal = {Journal of Endourology},
year = {2024},
volume = {38},
number = {8},
pages = {799--808},
doi = {10.1089/end.2023.0413}
}
EUVIMED