Performance of Language Models on the Family Medicine In-Training Exam
Family Medicine · 2024
DOI: 10.22454/fammed.2024.233738 ↗Kurz gefasst
Fachartikel in der Zeitschrift Family Medicine, 2024. Verfasst von Rana E Hanna und 3 weiteren Autor:innen. Beteiligte Einrichtung: Morsani College of Medicine, University of South Florida, Tampa, FL. Hauptthemen laut MEDLINE: Family Practice, Educational Measurement. Die Arbeit wurde bisher 12-mal zitiert (Crossref). Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background and Objectives: Artificial intelligence (AI), such as ChatGPT and Bard, has gained popularity as a tool in medical education. The use of AI in family medicine has not yet been assessed. The objective of this study is to compare the performance of three large language models (LLMs; ChatGPT 3.5, ChatGPT 4.0, and Google Bard) on the family medicine in-training exam (ITE).
Methods
The 193 multiple-choice questions of the 2022 ITE, written by the American Board of Family Medicine, were inputted in ChatGPT 3.5, ChatGPT 4.0, and Bard. The LLMs’ performance was then scored and scaled.
Results
ChatGPT 4.0 scored 167/193 (86.5%) with a scaled score of 730 out of 800. According to the Bayesian score predictor, ChatGPT 4.0 has a 100% chance of passing the family medicine board exam. ChatGPT 3.5 scored 66.3%, translating to a scaled score of 400 and an 88% chance of passing the family medicine board exam. Bard scored 64.2%, with a scaled score of 380 and an 85% chance of passing the boards. Compared to the national average of postgraduate year 3 residents, only ChatGPT 4.0 surpassed the residents’ mean of 68.4%.
Conclusions
ChatGPT 4.0 was the only LLM that outperformed the family medicine postgraduate year 3 residents’ national averages on the 2022 ITE, providing robust explanations and demonstrating its potential use in delivering background information on common medical concepts that appear on board exams.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Morsani College of Medicine, University of South Florida, Tampa, FL.
- Department of Medical Education, Morsani College of Medicine, University of South Florida, Tampa, FL.
- Department of Family Medicine, Morsani College of Medicine, University of South Florida, Tampa, FL.
Zitieren
APA
Hanna, R. E., Smith, L. R., Mhaskar, R., & Hanna, K. (2024). Performance of Language Models on the Family Medicine In-Training Exam. Family Medicine. https://doi.org/10.22454/fammed.2024.233738
Vancouver
Hanna RE, Smith LR, Mhaskar R, Hanna K. Performance of Language Models on the Family Medicine In-Training Exam. Family Medicine. 2024. doi:10.22454/fammed.2024.233738
BibTeX
@article{Hanna2024,
author = {Hanna, Rana E and Smith, Logan R and Mhaskar, Rahul and Hanna, Karim},
title = {Performance of Language Models on the Family Medicine In-Training Exam},
journal = {Family Medicine},
year = {2024},
doi = {10.22454/fammed.2024.233738}
}
EUVIMED