EUVIMEDEUVIMED entdecken
Vergleichsstudie

Rise of the Machines: Comparing Performance of Artificial Intelligence Large Language Models on Pharmacy Specialty Certification Examination Practice Questions

Curtis D Collins iD, Michael P Veve iD, Ian B Hollis iD

JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY · 2026

DOI: 10.1002/jac5.70215 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY, 2026. Verfasst von Curtis D Collins und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Pharmacy Services, Trinity Health Ann Arbor, Ann Arbor, Michigan, USA. Hauptthemen laut MEDLINE: Large Language Models, Certification, Artificial Intelligence, Educational Measurement. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 26 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) are increasingly used for clinical information retrieval and decision support, yet comparative performance on pharmacy board examination‐style content across specialties remains incompletely characterized.

Methods

We evaluated 15 LLMs using 145 publicly available Board of Pharmacy Specialties (BPS) certification practice questions spanning 14 specialty domains. Questions were entered using a standardized prompt without additional prompt engineering. Model responses were scored against BPS‐posted answer keys. Overall and specialty‐level accuracy were summarized descriptively. Differences among LLMs were tested using Cochran's Q with Bonferroni‐adjusted McNemar pairwise comparisons when appropriate, and LLMs were assessed using their default user‐facing settings.

Results

Across all LLMs, mean accuracy was 86.2% (standard deviation [SD], 3.5%), corresponding to an average of 125/145 items answered correctly. Accuracy ranged from 79.3% (95% confidence interval [CI], 72.6%–86%) for Perplexity AI to 91.7% (95% CI, 87.2%–96.3%) for Microsoft Copilot (GPT‐5). Overall performance differed significantly across LLMs (Cochran's Q = 46.262; df = 14; p < 0.001). After Bonferroni adjustment, Microsoft Copilot (GPT‐5), Google Gemini 2.5 Flash, and OpenAI o3 (Reasoning) outperformed Perplexity AI ( p < 0.001). Microsoft Copilot (GPT‐5) also outperformed an earlier version of Microsoft Copilot (GPT‐4.1) ( p < 0.001). Specialty‐level heterogeneity was generally limited, with significant model differences observed in Solid Organ Transplantation Pharmacy and Nuclear Pharmacy.

Conclusions

LLMs demonstrated high accuracy on BPS certification practice questions, with limited variability across LLMs and select specialty domains. These findings support continued evaluation of LLMs for potential use in pharmacy practice and clinical decision support, emphasizing the need for domain‐specific validation and ongoing monitoring as LLMs evolve.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsCertificationArtificial IntelligenceEducational MeasurementEducation, PharmacyHumans

Stichwörter der Autor:innen

artificial intelligenceboard certificationlarge language modelspharmacypharmacy education

Autor:innen und Institutionen

  • Department of Pharmacy Services, Trinity Health Ann Arbor, Ann Arbor, Michigan, USA.
  • Department of Pharmacy, Henry Ford Hospital, Detroit, Michigan, USA.
  • Department of Pharmacy Practice, Eugene Applebaum College of Pharmacy and Health Sciences, Wayne State University, Detroit, Michigan, USA.
  • University of North Carolina Eshelman School of Pharmacy, University of North Carolina, Chapel Hill, North Carolina, USA.

ORCID-Profile: Curtis D Collins, Michael P Veve, Ian B Hollis

Zitieren

APA

Collins, C. D., Veve, M. P., & Hollis, I. B. (2026). Rise of the Machines: Comparing Performance of Artificial Intelligence Large Language Models on Pharmacy Specialty Certification Examination Practice Questions. JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY. https://doi.org/10.1002/jac5.70215

Vancouver

Collins CD, Veve MP, Hollis IB. Rise of the Machines: Comparing Performance of Artificial Intelligence Large Language Models on Pharmacy Specialty Certification Examination Practice Questions. JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY. 2026. doi:10.1002/jac5.70215

BibTeX
@article{Collins2026,
  author = {Collins, Curtis D and Veve, Michael P and Hollis, Ian B},
  title = {Rise of the Machines: Comparing Performance of Artificial Intelligence Large Language Models on Pharmacy Specialty Certification Examination Practice Questions},
  journal = {JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY},
  year = {2026},
  doi = {10.1002/jac5.70215}
}
Kostenlos starten →