How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
JMIR Medical Education · 2023-02-08 · 9 · e45312
DOI: 10.2196/45312 ↗At a glance
Journal article in JMIR Medical Education, published online on 2023-02-08. Written by Aidan Gilson and 6 co-authors. Institution: Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, CT, United States. Cited 2053 times so far (Crossref); 28 references. The full text is freely available in PubMed Central. Funded by: NIDDK NIH HHS, NHLBI NIH HHS, NCATS NIH HHS.
Abstract (original language)
Background
Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input.
Objective
This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability.
Methods
We used 2 sets of multiple-choice questions to evaluate ChatGPT’s performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT’s performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question.
Results
Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT’s answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P
Subjects
Author keywords
Authors and institutions
- Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, CT, United States.
- Department of Emergency Medicine, Yale University School of Medicine, New Haven, CT, United States.
- Program of Computational Biology and Bioinformatics, Yale University, New Haven, CT, United States.
- School of Medicine, University College Dublin, National University of Ireland, Dublin, Dublin, Ireland.
ORCID profiles: Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash
Funding
- NIDDK NIH HHS
- NHLBI NIH HHS
- NCATS NIH HHS
Cite
APA
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023). How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Medical Education, 9, e45312. https://doi.org/10.2196/45312
Vancouver
Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Medical Education. 2023;9:e45312. doi:10.2196/45312
BibTeX
@article{Gilson2023,
author = {Gilson, Aidan and Safranek, Conrad W and Huang, Thomas and Socrates, Vimig and Chi, Ling and Taylor, Richard Andrew and Chartash, David},
title = {How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment},
journal = {JMIR Medical Education},
year = {2023},
volume = {9},
pages = {e45312},
doi = {10.2196/45312}
}
EUVIMED