Analysis of ChatGPT-3.5’s Potential in Generating NBME-Standard Pharmacology Questions: What Can Be Improved?
Algorithms · 21.10.2024 · 17 (10) · 469
DOI: 10.3390/a17100469 ↗Kurz gefasst
Fachartikel in der Zeitschrift Algorithms, online erschienen am 21.10.2024. Verfasst von Marwa Saad und 4 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
ChatGPT by OpenAI is an AI model designed to generate human-like responses based on diverse datasets. Our study evaluated ChatGPT-3.5’s capability to generate pharmacology multiple-choice questions adhering to the NBME guidelines for USMLE Step exams. The initial findings show ChatGPT’s rapid adoption and potential in healthcare education and practice. However, concerns about its accuracy and depth of understanding prompted this evaluation. Using a structured prompt engineering process, ChatGPT was tasked to generate questions across various organ systems, which were then reviewed by pharmacology experts. ChatGPT consistently met the NBME criteria, achieving an average score of 13.7 out of 16 (85.6%) from expert 1 and 14.5 out of 16 (90.6%) from expert 2, with a combined average of 14.1 out of 16 (88.1%) (Kappa coefficient = 0.76). Despite these high scores, challenges in medical accuracy and depth were noted, often producing “pseudo vignettes” instead of in-depth clinical questions. ChatGPT-3.5 shows potential for generating NBME-style questions, but improvements in medical accuracy and understanding are crucial for its reliable use in medical education. This study underscores the need for AI models tailored to the medical domain to enhance educational tools for medical students.
Zitieren
APA
Saad, M., Almasri, W., Hye, T., Roni, M., & Mohiyeddini, C. (2024). Analysis of ChatGPT-3.5’s Potential in Generating NBME-Standard Pharmacology Questions: What Can Be Improved?. Algorithms, 17(10), 469. https://doi.org/10.3390/a17100469
Vancouver
Saad M, Almasri W, Hye T, Roni M, Mohiyeddini C. Analysis of ChatGPT-3.5’s Potential in Generating NBME-Standard Pharmacology Questions: What Can Be Improved?. Algorithms. 2024;17(10):469. doi:10.3390/a17100469
BibTeX
@article{Saad2024,
author = {Saad, Marwa and Almasri, Wesam and Hye, Tanvirul and Roni, Monzurul and Mohiyeddini, Changiz},
title = {Analysis of ChatGPT-3.5’s Potential in Generating NBME-Standard Pharmacology Questions: What Can Be Improved?},
journal = {Algorithms},
year = {2024},
volume = {17},
number = {10},
pages = {469},
doi = {10.3390/a17100469}
}
EUVIMED