EUVIMEDDécouvrir EUVIMED
Systematische ÜbersichtMetaanalyseOpen access · CC BY-NC-ND 4.0

Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis

Hye Kyung Jin, Ha Eun Lee, EunYoung Kim

BMC Medical Education · 2024

DOI: 10.1186/s12909-024-05944-8 ↗

At a glance

Systematische Übersicht in BMC Medical Education, 2024. Written by Hye Kyung Jin and 2 co-authors. Institution: Research Institute of Pharmaceutical Sciences, College of Pharmacy, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul, 06974, South Korea. Major MEDLINE topics: Artificial Intelligence, Educational Measurement, Licensure. Cited 86 times so far (Crossref); 59 references. The full text is freely available (open access, CC BY-NC-ND 4.0). Funded by: the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Chung-Ang University Graduate Research Scholarship in 2023.

RésuméSubjectsRéférences bibliographiquesCiteRelated studies

Abstract (original language)

BACKGROUND: ChatGPT, a recently developed artificial intelligence (AI) chatbot, has demonstrated improved performance in examinations in the medical field. However, thus far, an overall evaluation of the potential of ChatGPT models (ChatGPT-3.5 and GPT-4) in a variety of national health licensing examinations is lacking. This study aimed to provide a comprehensive assessment of the ChatGPT models' performance in national licensing examinations for medical, pharmacy, dentistry, and nursing research through a meta-analysis. METHODS: Following the PRISMA protocol, full-text articles from MEDLINE/PubMed, EMBASE, ERIC, Cochrane Library, Web of Science, and key journals were reviewed from the time of ChatGPT's introduction to February 27, 2024. Studies were eligible if they evaluated the performance of a ChatGPT model (ChatGPT-3.5 or GPT-4); related to national licensing examinations in the fields of medicine, pharmacy, dentistry, or nursing; involved multiple-choice questions; and provided data that enabled the calculation of effect size. Two reviewers independently completed data extraction, coding, and quality assessment. The JBI Critical Appraisal Tools were used to assess the quality of the selected articles. Overall effect size and 95% confidence intervals [CIs] were calculated using a random-effects model. RESULTS: A total of 23 studies were considered for this review, which evaluated the accuracy of four types of national licensing examinations. The selected articles were in the fields of medicine (n = 17), pharmacy (n = 3), nursing (n = 2), and dentistry (n = 1). They reported varying accuracy levels, ranging from 36 to 77% for ChatGPT-3.5 and 64.4-100% for GPT-4. The overall effect size for the percentage of accuracy was 70.1% (95% CI, 65-74.8%), which was statistically significant (p < 0.001). Subgroup analyses revealed that GPT-4 demonstrated significantly higher accuracy in providing correct responses than its earlier version, ChatGPT-3.5. Additionally, in the context of health licensing examinations, the ChatGPT models exhibited greater proficiency in the following order: pharmacy, medicine, dentistry, and nursing. However, the lack of a broader set of questions, including open-ended and scenario-based questions, and significant heterogeneity were limitations of this meta-analysis. CONCLUSIONS: This study sheds light on the accuracy of ChatGPT models in four national health licensing examinations across various countries and provides a practical basis and theoretical support for future research. Further studies are needed to explore their utilization in medical and health education by including a broader and more diverse range of questions, along with more advanced versions of AI chatbots.

Subjects

MeSH terms (MEDLINE)

Artificial IntelligenceEducational MeasurementLicensureHumansEducation, NursingEducation, PharmacyEducation, MedicalEducation, Dental

Author keywords

ChatGPT-3.5GPT-4Healthcare professionalsMeta-analysisNational licensing examination

Authors and institutions

  • Research Institute of Pharmaceutical Sciences, College of Pharmacy, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul, 06974, South Korea.
  • Data Science, Evidence-Based and Clinical Research Laboratory, Department of Health, Social, and Clinical Pharmacy, College of Pharmacy, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul, 06974, South Korea.
  • Research Institute of Pharmaceutical Sciences, College of Pharmacy, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul, 06974, South Korea. eykimjcb777@cau.ac.kr.
  • Data Science, Evidence-Based and Clinical Research Laboratory, Department of Health, Social, and Clinical Pharmacy, College of Pharmacy, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul, 06974, South Korea. eykim…
  • Division of Licensing of Medicines and Regulatory Science, The Graduate School of Pharmaceutical Management, and Regulatory Science Policy, The Graduate School of Pharmaceutical Regulatory Sciences, Chung-Ang University…

Funding

  • the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education
  • Chung-Ang University Graduate Research Scholarship in 2023

Cite

APA

Jin, H. K., Lee, H. E., & Kim, E. (2024). Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis. BMC Medical Education. https://doi.org/10.1186/s12909-024-05944-8

Vancouver

Jin HK, Lee HE, Kim E. Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis. BMC Medical Education. 2024. doi:10.1186/s12909-024-05944-8

BibTeX
@article{Jin2024,
  author = {Jin, Hye Kyung and Lee, Ha Eun and Kim, EunYoung},
  title = {Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis},
  journal = {BMC Medical Education},
  year = {2024},
  doi = {10.1186/s12909-024-05944-8}
}
Start for free →