Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical Examination: a cross-sectional study
Journal of Educational Evaluation for Health Professions · 20.11.2023 · 20 · 30
DOI: 10.3352/jeehp.2023.20.30 ↗Kurz gefasst
Querschnittstudie in der Zeitschrift Journal of Educational Evaluation for Health Professions, online erschienen am 20.11.2023. Verfasst von Betzy Clariza Torres-Zegarra und 10 weiteren Autor:innen. Beteiligte Einrichtung: Escuela de Medicina, Universidad Cientifica del Sur, Lima, Peru. Hauptthemen laut MEDLINE: Artificial Intelligence, Knowledge. Die Arbeit wurde bisher 62-mal zitiert (Crossref) und nennt 13 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Purpose
We aimed to describe the performance and evaluate the educational value of justifications provided by artificial intelligence chatbots, including GPT-3.5, GPT-4, Bard, Claude, and Bing, on the Peruvian National Medical Licensing Examination (P-NLME).
Methods
This was a cross-sectional analytical study. On July 25, 2023, each multiple-choice question (MCQ) from the P-NLME was entered into each chatbot (GPT-3, GPT-4, Bing, Bard, and Claude) 3 times. Then, 4 medical educators categorized the MCQs in terms of medical area, item type, and whether the MCQ required Peru-specific knowledge. They assessed the educational value of the justifications from the 2 top performers (GPT-4 and Bing).
Results
GPT-4 scored 86.7% and Bing scored 82.2%, followed by Bard and Claude, and the historical performance of Peruvian examinees was 55%. Among the factors associated with correct answers, only MCQs that required Peru-specific knowledge had lower odds (odds ratio, 0.23; 95% confidence interval, 0.09–0.61), whereas the remaining factors showed no associations. In assessing the educational value of justifications provided by GPT-4 and Bing, neither showed any significant differences in certainty, usefulness, or potential use in the classroom.
Conclusion
Among chatbots, GPT-4 and Bing were the top performers, with Bing performing better at Peru-specific MCQs. Moreover, the educational value of justifications provided by the GPT-4 and Bing could be deemed appropriate. However, it is essential to start addressing the educational value of these chatbots, rather than merely their performance on examinations.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Escuela de Medicina, Universidad Cientifica del Sur, Lima, Peru.
- Sociedad Científica de Estudiantes de Medicina de Ica, Universidad Nacional San Luis Gonzaga, Ica, Peru.
- Universidad Nacional de Cajamarca, Cajamarca, Peru.
- Academic Department, USAMEDIC, Lima, Peru.
- Neurogenetics Research Center, Instituto Nacional de Ciencias Neurologicas, Lima, Peru.
Zitieren
APA
Torres-Zegarra, B. C., Rios-Garcia, W., Ñaña-Cordova, A. M., Arteaga-Cisneros, K. F., Chalco, X. C. B., Ordoñez, M. A. B., Rios, C. J. G., Godoy, C. A. R., Quezada, K. L. T. P., Gutierrez-Arratia, J. D., & Flores-Cohaila, J. A. (2023). Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical Examination: a cross-sectional study. Journal of Educational Evaluation for Health Professions, 20, 30. https://doi.org/10.3352/jeehp.2023.20.30
Vancouver
Torres-Zegarra BC, Rios-Garcia W, Ñaña-Cordova AM, Arteaga-Cisneros KF, Chalco XCB, Ordoñez MAB, et al. Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical Examination: a cross-sectional study. Journal of Educational Evaluation for Health Professions. 2023;20:30. doi:10.3352/jeehp.2023.20.30
BibTeX
@article{TorresZegarra2023,
author = {Torres-Zegarra, Betzy Clariza and Rios-Garcia, Wagner and Ñaña-Cordova, Alvaro Micael and Arteaga-Cisneros, Karen Fatima and Chalco, Xiomara Cristina Benavente and Ordoñez, Marina Atena Bustamante and Rios, Carlos Jesus Gutierrez and Godoy, Carlos Alberto Ramos and Quezada, Kristell Luisa Teresa Panta and Gutierrez-Arratia, Jesus Daniel and Flores-Cohaila, Javier Alejandro},
title = {Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical Examination: a cross-sectional study},
journal = {Journal of Educational Evaluation for Health Professions},
year = {2023},
volume = {20},
pages = {30},
doi = {10.3352/jeehp.2023.20.30}
}
EUVIMED