Performance of DeepSeek and GPT Models on Pediatric Board Preparation Questions: Comparative Evaluation
JMIR AI · 27.08.2025 · 4 · e76056-e76056
DOI: 10.2196/76056 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR AI, online erschienen am 27.08.2025. Verfasst von Masab Mansoor und 2 weiteren Autor:innen. Beteiligte Einrichtung: Louisiana Campus, Edward Via College of Osteopathic Medicine, 4408 Bon Aire Dr, Monroe, LA, 71203, United States, 1 5045213500. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 13 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Limited research exists evaluating artificial intelligence (AI) performance on standardized pediatric assessments. This study evaluated 3 leading AI models on pediatric board preparation questions.
Objective
The aim of this study is to evaluate and compare the performance of 3 leading large language models (LLMs) on pediatric board examination preparation questions and contextualize their performance against human physician benchmarks.
Methods
We analyzed DeepSeek-R1, ChatGPT-4, and ChatGPT-4.5 using 266 multiple-choice questions from the 2023 PREP Self-Assessment. Performance was compared to published American Board of Pediatrics first-time pass rates.
Results
DeepSeek-R1 exhibited the highest accuracy at 98.1% (261/266 correct responses). ChatGPT-4.5 achieved 96.6% accuracy (257/266), performing at the upper threshold of human performance. ChatGPT-4 demonstrated 82.7% accuracy (220/266), comparable to the lower range of human pass rates. Error pattern analysis revealed that AI models most commonly struggled with questions requiring integration of complex clinical presentations with rare disease knowledge.
Conclusions
DeepSeek-R1 demonstrated exceptional performance exceeding typical American Board of Pediatrics pass rates, suggesting potential applications in medical education and clinical support, though further research on complex clinical reasoning is needed.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Louisiana Campus, Edward Via College of Osteopathic Medicine, 4408 Bon Aire Dr, Monroe, LA, 71203, United States, 1 5045213500.
- School of Medicine, Texas Tech University Health Sciences Center, Lubbock, TX, United States.
Zitieren
APA
Mansoor, M., Ibrahim, A., & Hamide, A. (2025). Performance of DeepSeek and GPT Models on Pediatric Board Preparation Questions: Comparative Evaluation. JMIR AI, 4, e76056-e76056. https://doi.org/10.2196/76056
Vancouver
Mansoor M, Ibrahim A, Hamide A. Performance of DeepSeek and GPT Models on Pediatric Board Preparation Questions: Comparative Evaluation. JMIR AI. 2025;4:e76056-e76056. doi:10.2196/76056
BibTeX
@article{Mansoor2025,
author = {Mansoor, Masab and Ibrahim, Andrew and Hamide, Ali},
title = {Performance of DeepSeek and GPT Models on Pediatric Board Preparation Questions: Comparative Evaluation},
journal = {JMIR AI},
year = {2025},
volume = {4},
pages = {e76056--e76056},
doi = {10.2196/76056}
}
EUVIMED