EUVIMEDEUVIMED entdecken
Crossref

Performance of large language models on family medicine licensing exams

Mahmud Omar iD, Kareem Hijazi, Mohammad Omar, Girish N Nadkarni, Eyal Klang

Family Practice · 2025

DOI: 10.1093/fampra/cmaf035 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Family Practice, 2025. Verfasst von Mahmud Omar und 4 weiteren Autor:innen. Beteiligte Einrichtung: The Windreich Department of Artificial Intelligence and Human Health, Mount Sinai Medical Center, New York, United States. Hauptthemen laut MEDLINE: Family Practice, Educational Measurement, Language, Licensure, Medical. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 19 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background and Aim Large language models (LLMs) have shown promise in specialized medical exams but remain less explored in family medicine and primary care. This study evaluated eight state-of-the-art LLMs on the official Israeli primary care licensing exam, focusing on prompt design and explanation quality.

Methods

Two hundred multiple-choice questions were tested using simple and few-shot Chain-of-Thought prompts (prompts that include examples which illustrate reasoning). Performance differences were assessed with Cochran’s Q and pairwise McNemar tests. A stress test of the top performer (openAI’s o1-preview) examined 30 selected questions, with two physicians scoring explanations for accuracy, logic, and hallucinations (extra or fabricated information not supported by the question).

Results

Five models exceeded the 65% passing threshold under simple prompts; seven did so with few-shot prompts. o1-preview reached 85.5%. In the stress test, explanations were generally coherent and accurate, with 5 of 120 flagged for hallucinations. Inter-rater agreement on explanation scoring was high (weighted kappa 0.773; Intraclass Correlation Coefficient (ICC) 0.776).

Conclusions

Most tested models performed well on an official family medicine exam, especially with structured prompts. Nonetheless, multiple-choice formats cannot address broader clinical competencies such as physical exams and patient rapport. Future efforts should refine these models to eliminate hallucinations, test for socio-demographic biases, and ensure alignment with real-world demands.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Family PracticeEducational MeasurementLanguageLicensure, MedicalHumansIsraelPrimary Health CareClinical CompetenceLarge Language Models

Stichwörter der Autor:innen

artificial intelligencelarge language modelslicensing examprimary careprompt engineering

Autor:innen und Institutionen

  • The Windreich Department of Artificial Intelligence and Human Health, Mount Sinai Medical Center, New York, United States.
  • Maccabi Healthcare Services, Tel-Aviv, Israel.
  • Emek Medical Ce6nter, Afula, Israel.

ORCID-Profile: Mahmud Omar

Zitieren

APA

Omar, M., Hijazi, K., Omar, M., Nadkarni, G. N., & Klang, E. (2025). Performance of large language models on family medicine licensing exams. Family Practice. https://doi.org/10.1093/fampra/cmaf035

Vancouver

Omar M, Hijazi K, Omar M, Nadkarni GN, Klang E. Performance of large language models on family medicine licensing exams. Family Practice. 2025. doi:10.1093/fampra/cmaf035

BibTeX
@article{Omar2025,
  author = {Omar, Mahmud and Hijazi, Kareem and Omar, Mohammad and Nadkarni, Girish N and Klang, Eyal},
  title = {Performance of large language models on family medicine licensing exams},
  journal = {Family Practice},
  year = {2025},
  doi = {10.1093/fampra/cmaf035}
}
Kostenlos starten →