EUVIMEDEUVIMED entdecken
QuerschnittstudieOpen Access · CC BY 4.0

Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical Examination: a cross-sectional study

Betzy Clariza Torres-Zegarra, Wagner Rios-Garcia, Alvaro Micael Ñaña-Cordova, Karen Fatima Arteaga-Cisneros, Xiomara Cristina Benavente Chalco, Marina Atena Bustamante Ordoñez, Carlos Jesus Gutierrez Rios, Carlos Alberto Ramos Godoy, Kristell Luisa Teresa Panta Quezada, Jesus Daniel Gutierrez-Arratia, Javier Alejandro Flores-Cohaila

Journal of Educational Evaluation for Health Professions · 20.11.2023 · 20 · 30

DOI: 10.3352/jeehp.2023.20.30 ↗

Kurz gefasst

Querschnittstudie in der Zeitschrift Journal of Educational Evaluation for Health Professions, online erschienen am 20.11.2023. Verfasst von Betzy Clariza Torres-Zegarra und 10 weiteren Autor:innen. Beteiligte Einrichtung: Escuela de Medicina, Universidad Cientifica del Sur, Lima, Peru. Hauptthemen laut MEDLINE: Artificial Intelligence, Knowledge. Die Arbeit wurde bisher 62-mal zitiert (Crossref) und nennt 13 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

We aimed to describe the performance and evaluate the educational value of justifications provided by artificial intelligence chatbots, including GPT-3.5, GPT-4, Bard, Claude, and Bing, on the Peruvian National Medical Licensing Examination (P-NLME).

Methods

This was a cross-sectional analytical study. On July 25, 2023, each multiple-choice question (MCQ) from the P-NLME was entered into each chatbot (GPT-3, GPT-4, Bing, Bard, and Claude) 3 times. Then, 4 medical educators categorized the MCQs in terms of medical area, item type, and whether the MCQ required Peru-specific knowledge. They assessed the educational value of the justifications from the 2 top performers (GPT-4 and Bing).

Results

GPT-4 scored 86.7% and Bing scored 82.2%, followed by Bard and Claude, and the historical performance of Peruvian examinees was 55%. Among the factors associated with correct answers, only MCQs that required Peru-specific knowledge had lower odds (odds ratio, 0.23; 95% confidence interval, 0.09–0.61), whereas the remaining factors showed no associations. In assessing the educational value of justifications provided by GPT-4 and Bing, neither showed any significant differences in certainty, usefulness, or potential use in the classroom.

Conclusion

Among chatbots, GPT-4 and Bing were the top performers, with Bing performing better at Peru-specific MCQs. Moreover, the educational value of justifications provided by the GPT-4 and Bing could be deemed appropriate. However, it is essential to start addressing the educational value of these chatbots, rather than merely their performance on examinations.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceKnowledgeHumansCross-Sectional StudiesPeruEducational Status

Stichwörter der Autor:innen

Artificial intelligenceEducational measurementMedical educationPeru

Autor:innen und Institutionen

  • Escuela de Medicina, Universidad Cientifica del Sur, Lima, Peru.
  • Sociedad Científica de Estudiantes de Medicina de Ica, Universidad Nacional San Luis Gonzaga, Ica, Peru.
  • Universidad Nacional de Cajamarca, Cajamarca, Peru.
  • Academic Department, USAMEDIC, Lima, Peru.
  • Neurogenetics Research Center, Instituto Nacional de Ciencias Neurologicas, Lima, Peru.

Zitieren

APA

Torres-Zegarra, B. C., Rios-Garcia, W., Ñaña-Cordova, A. M., Arteaga-Cisneros, K. F., Chalco, X. C. B., Ordoñez, M. A. B., Rios, C. J. G., Godoy, C. A. R., Quezada, K. L. T. P., Gutierrez-Arratia, J. D., & Flores-Cohaila, J. A. (2023). Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical Examination: a cross-sectional study. Journal of Educational Evaluation for Health Professions, 20, 30. https://doi.org/10.3352/jeehp.2023.20.30

Vancouver

Torres-Zegarra BC, Rios-Garcia W, Ñaña-Cordova AM, Arteaga-Cisneros KF, Chalco XCB, Ordoñez MAB, et al. Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical Examination: a cross-sectional study. Journal of Educational Evaluation for Health Professions. 2023;20:30. doi:10.3352/jeehp.2023.20.30

BibTeX
@article{TorresZegarra2023,
  author = {Torres-Zegarra, Betzy Clariza and Rios-Garcia, Wagner and Ñaña-Cordova, Alvaro Micael and Arteaga-Cisneros, Karen Fatima and Chalco, Xiomara Cristina Benavente and Ordoñez, Marina Atena Bustamante and Rios, Carlos Jesus Gutierrez and Godoy, Carlos Alberto Ramos and Quezada, Kristell Luisa Teresa Panta and Gutierrez-Arratia, Jesus Daniel and Flores-Cohaila, Javier Alejandro},
  title = {Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical Examination: a cross-sectional study},
  journal = {Journal of Educational Evaluation for Health Professions},
  year = {2023},
  volume = {20},
  pages = {30},
  doi = {10.3352/jeehp.2023.20.30}
}
Kostenlos starten →