EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Analysis of ChatGPT-3.5’s Potential in Generating NBME-Standard Pharmacology Questions: What Can Be Improved?

Marwa Saad, Wesam Almasri, Tanvirul Hye, Monzurul Roni, Changiz Mohiyeddini

Algorithms · 21.10.2024 · 17 (10) · 469

DOI: 10.3390/a17100469 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Algorithms, online erschienen am 21.10.2024. Verfasst von Marwa Saad und 4 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

ChatGPT by OpenAI is an AI model designed to generate human-like responses based on diverse datasets. Our study evaluated ChatGPT-3.5’s capability to generate pharmacology multiple-choice questions adhering to the NBME guidelines for USMLE Step exams. The initial findings show ChatGPT’s rapid adoption and potential in healthcare education and practice. However, concerns about its accuracy and depth of understanding prompted this evaluation. Using a structured prompt engineering process, ChatGPT was tasked to generate questions across various organ systems, which were then reviewed by pharmacology experts. ChatGPT consistently met the NBME criteria, achieving an average score of 13.7 out of 16 (85.6%) from expert 1 and 14.5 out of 16 (90.6%) from expert 2, with a combined average of 14.1 out of 16 (88.1%) (Kappa coefficient = 0.76). Despite these high scores, challenges in medical accuracy and depth were noted, often producing “pseudo vignettes” instead of in-depth clinical questions. ChatGPT-3.5 shows potential for generating NBME-style questions, but improvements in medical accuracy and understanding are crucial for its reliable use in medical education. This study underscores the need for AI models tailored to the medical domain to enhance educational tools for medical students.

Zitieren

APA

Saad, M., Almasri, W., Hye, T., Roni, M., & Mohiyeddini, C. (2024). Analysis of ChatGPT-3.5’s Potential in Generating NBME-Standard Pharmacology Questions: What Can Be Improved?. Algorithms, 17(10), 469. https://doi.org/10.3390/a17100469

Vancouver

Saad M, Almasri W, Hye T, Roni M, Mohiyeddini C. Analysis of ChatGPT-3.5’s Potential in Generating NBME-Standard Pharmacology Questions: What Can Be Improved?. Algorithms. 2024;17(10):469. doi:10.3390/a17100469

BibTeX
@article{Saad2024,
  author = {Saad, Marwa and Almasri, Wesam and Hye, Tanvirul and Roni, Monzurul and Mohiyeddini, Changiz},
  title = {Analysis of ChatGPT-3.5’s Potential in Generating NBME-Standard Pharmacology Questions: What Can Be Improved?},
  journal = {Algorithms},
  year = {2024},
  volume = {17},
  number = {10},
  pages = {469},
  doi = {10.3390/a17100469}
}
Kostenlos starten →