Performance of large language models on family medicine licensing exams
Family Practice · 2025
DOI: 10.1093/fampra/cmaf035 ↗Kurz gefasst
Fachartikel in der Zeitschrift Family Practice, 2025. Verfasst von Mahmud Omar und 4 weiteren Autor:innen. Beteiligte Einrichtung: The Windreich Department of Artificial Intelligence and Human Health, Mount Sinai Medical Center, New York, United States. Hauptthemen laut MEDLINE: Family Practice, Educational Measurement, Language, Licensure, Medical. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 19 Literaturangaben.
Abstract (Originalsprache)
Background and Aim Large language models (LLMs) have shown promise in specialized medical exams but remain less explored in family medicine and primary care. This study evaluated eight state-of-the-art LLMs on the official Israeli primary care licensing exam, focusing on prompt design and explanation quality.
Methods
Two hundred multiple-choice questions were tested using simple and few-shot Chain-of-Thought prompts (prompts that include examples which illustrate reasoning). Performance differences were assessed with Cochran’s Q and pairwise McNemar tests. A stress test of the top performer (openAI’s o1-preview) examined 30 selected questions, with two physicians scoring explanations for accuracy, logic, and hallucinations (extra or fabricated information not supported by the question).
Results
Five models exceeded the 65% passing threshold under simple prompts; seven did so with few-shot prompts. o1-preview reached 85.5%. In the stress test, explanations were generally coherent and accurate, with 5 of 120 flagged for hallucinations. Inter-rater agreement on explanation scoring was high (weighted kappa 0.773; Intraclass Correlation Coefficient (ICC) 0.776).
Conclusions
Most tested models performed well on an official family medicine exam, especially with structured prompts. Nonetheless, multiple-choice formats cannot address broader clinical competencies such as physical exams and patient rapport. Future efforts should refine these models to eliminate hallucinations, test for socio-demographic biases, and ensure alignment with real-world demands.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- The Windreich Department of Artificial Intelligence and Human Health, Mount Sinai Medical Center, New York, United States.
- Maccabi Healthcare Services, Tel-Aviv, Israel.
- Emek Medical Ce6nter, Afula, Israel.
ORCID-Profile: Mahmud Omar
Zitieren
APA
Omar, M., Hijazi, K., Omar, M., Nadkarni, G. N., & Klang, E. (2025). Performance of large language models on family medicine licensing exams. Family Practice. https://doi.org/10.1093/fampra/cmaf035
Vancouver
Omar M, Hijazi K, Omar M, Nadkarni GN, Klang E. Performance of large language models on family medicine licensing exams. Family Practice. 2025. doi:10.1093/fampra/cmaf035
BibTeX
@article{Omar2025,
author = {Omar, Mahmud and Hijazi, Kareem and Omar, Mohammad and Nadkarni, Girish N and Klang, Eyal},
title = {Performance of large language models on family medicine licensing exams},
journal = {Family Practice},
year = {2025},
doi = {10.1093/fampra/cmaf035}
}
EUVIMED