ChatGPT's Medical Exam Performance: Version and Language Analysis in General Surgery Fellowship Exam
European Journal of Human Health · 2025 · 1 (1) · 1-9
DOI: 10.29228/ejhh.80681 ↗Kurz gefasst
Fachartikel in der Zeitschrift European Journal of Human Health, 2025. Verfasst von Süleyman Orman. Die Arbeit wurde bisher 2-mal zitiert (Crossref).
Abstract (Originalsprache)
Objective
The integration of Artificial Intelligence (AI) in medical education has the potential to revolutionize learning and assessment.This study evaluates the performance of ChatGPT-3.5 and ChatGPT-4 on the General Surgery Fellowship Examination (GSFE) in Turkey, comparing their accuracy in answering multiple-choice questions (MCQs) in Turkish and English.
Methods
255 retired and publicly available GSFE questions (2011-2022) were analyzed.Questions were first presented in Turkish and subsequently translated into English for re-evaluation.ChatGPT-3.5 and ChatGPT-4 were prompted as if they were general surgeons answering the MCQs.The accuracy of responses was assessed, and statistical analyses were performed to identify significant differences between the bots and languages.
Results
In Turkish, ChatGPT-3.5 achieved 66.66% accuracy (170/255 correct answers), while ChatGPT-4 scored 69.41% (177/255).In English, ChatGPT-3.5 achieved 67.05% accuracy (171/255), and ChatGPT-4 scored 70.19% (179/255).Statistically significant differences were observed between ChatGPT-3.5 and ChatGPT-4 for both Turkish (p 0.05).It was found that Cohen's Kappa number was quite high between pairwise comparisons between applications, and the consistency among the four applications showed similarly high agreement (Fleiss' Kappa = 0.95, p
Zitieren
APA
Orman, S. (2025). ChatGPT's Medical Exam Performance: Version and Language Analysis in General Surgery Fellowship Exam. European Journal of Human Health, 1(1), 1-9. https://doi.org/10.29228/ejhh.80681
Vancouver
Orman S. ChatGPT's Medical Exam Performance: Version and Language Analysis in General Surgery Fellowship Exam. European Journal of Human Health. 2025;1(1):1-9. doi:10.29228/ejhh.80681
BibTeX
@article{Orman2025,
author = {Orman, Süleyman},
title = {ChatGPT's Medical Exam Performance: Version and Language Analysis in General Surgery Fellowship Exam},
journal = {European Journal of Human Health},
year = {2025},
volume = {1},
number = {1},
pages = {1--9},
doi = {10.29228/ejhh.80681}
}
EUVIMED