Performance of DeepSeek and GPT Models on Pediatric Board Preparation Questions: Comparative Evaluation
JMIR AI · 2025-08-27 · 4 · e76056-e76056
DOI: 10.2196/76056 ↗At a glance
Journal article in JMIR AI, published online on 2025-08-27. Written by Masab Mansoor and 2 co-authors. Institution: Louisiana Campus, Edward Via College of Osteopathic Medicine, 4408 Bon Aire Dr, Monroe, LA, 71203, United States, 1 5045213500. Cited 9 times so far (Crossref); 13 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Limited research exists evaluating artificial intelligence (AI) performance on standardized pediatric assessments. This study evaluated 3 leading AI models on pediatric board preparation questions.
Objective
The aim of this study is to evaluate and compare the performance of 3 leading large language models (LLMs) on pediatric board examination preparation questions and contextualize their performance against human physician benchmarks.
Methods
We analyzed DeepSeek-R1, ChatGPT-4, and ChatGPT-4.5 using 266 multiple-choice questions from the 2023 PREP Self-Assessment. Performance was compared to published American Board of Pediatrics first-time pass rates.
Results
DeepSeek-R1 exhibited the highest accuracy at 98.1% (261/266 correct responses). ChatGPT-4.5 achieved 96.6% accuracy (257/266), performing at the upper threshold of human performance. ChatGPT-4 demonstrated 82.7% accuracy (220/266), comparable to the lower range of human pass rates. Error pattern analysis revealed that AI models most commonly struggled with questions requiring integration of complex clinical presentations with rare disease knowledge.
Conclusions
DeepSeek-R1 demonstrated exceptional performance exceeding typical American Board of Pediatrics pass rates, suggesting potential applications in medical education and clinical support, though further research on complex clinical reasoning is needed.
Subjects
Author keywords
Authors and institutions
- Louisiana Campus, Edward Via College of Osteopathic Medicine, 4408 Bon Aire Dr, Monroe, LA, 71203, United States, 1 5045213500.
- School of Medicine, Texas Tech University Health Sciences Center, Lubbock, TX, United States.
Cite
APA
Mansoor, M., Ibrahim, A., & Hamide, A. (2025). Performance of DeepSeek and GPT Models on Pediatric Board Preparation Questions: Comparative Evaluation. JMIR AI, 4, e76056-e76056. https://doi.org/10.2196/76056
Vancouver
Mansoor M, Ibrahim A, Hamide A. Performance of DeepSeek and GPT Models on Pediatric Board Preparation Questions: Comparative Evaluation. JMIR AI. 2025;4:e76056-e76056. doi:10.2196/76056
BibTeX
@article{Mansoor2025,
author = {Mansoor, Masab and Ibrahim, Andrew and Hamide, Ali},
title = {Performance of DeepSeek and GPT Models on Pediatric Board Preparation Questions: Comparative Evaluation},
journal = {JMIR AI},
year = {2025},
volume = {4},
pages = {e76056--e76056},
doi = {10.2196/76056}
}
EUVIMED