Rise of the Machines: Comparing Performance of Artificial Intelligence Large Language Models on Pharmacy Specialty Certification Examination Practice Questions
JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY · 2026
DOI: 10.1002/jac5.70215 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY, 2026. Verfasst von Curtis D Collins und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Pharmacy Services, Trinity Health Ann Arbor, Ann Arbor, Michigan, USA. Hauptthemen laut MEDLINE: Large Language Models, Certification, Artificial Intelligence, Educational Measurement. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 26 Literaturangaben.
Abstract (Originalsprache)
Background
Large language models (LLMs) are increasingly used for clinical information retrieval and decision support, yet comparative performance on pharmacy board examination‐style content across specialties remains incompletely characterized.
Methods
We evaluated 15 LLMs using 145 publicly available Board of Pharmacy Specialties (BPS) certification practice questions spanning 14 specialty domains. Questions were entered using a standardized prompt without additional prompt engineering. Model responses were scored against BPS‐posted answer keys. Overall and specialty‐level accuracy were summarized descriptively. Differences among LLMs were tested using Cochran's Q with Bonferroni‐adjusted McNemar pairwise comparisons when appropriate, and LLMs were assessed using their default user‐facing settings.
Results
Across all LLMs, mean accuracy was 86.2% (standard deviation [SD], 3.5%), corresponding to an average of 125/145 items answered correctly. Accuracy ranged from 79.3% (95% confidence interval [CI], 72.6%–86%) for Perplexity AI to 91.7% (95% CI, 87.2%–96.3%) for Microsoft Copilot (GPT‐5). Overall performance differed significantly across LLMs (Cochran's Q = 46.262; df = 14; p < 0.001). After Bonferroni adjustment, Microsoft Copilot (GPT‐5), Google Gemini 2.5 Flash, and OpenAI o3 (Reasoning) outperformed Perplexity AI ( p < 0.001). Microsoft Copilot (GPT‐5) also outperformed an earlier version of Microsoft Copilot (GPT‐4.1) ( p < 0.001). Specialty‐level heterogeneity was generally limited, with significant model differences observed in Solid Organ Transplantation Pharmacy and Nuclear Pharmacy.
Conclusions
LLMs demonstrated high accuracy on BPS certification practice questions, with limited variability across LLMs and select specialty domains. These findings support continued evaluation of LLMs for potential use in pharmacy practice and clinical decision support, emphasizing the need for domain‐specific validation and ongoing monitoring as LLMs evolve.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Pharmacy Services, Trinity Health Ann Arbor, Ann Arbor, Michigan, USA.
- Department of Pharmacy, Henry Ford Hospital, Detroit, Michigan, USA.
- Department of Pharmacy Practice, Eugene Applebaum College of Pharmacy and Health Sciences, Wayne State University, Detroit, Michigan, USA.
- University of North Carolina Eshelman School of Pharmacy, University of North Carolina, Chapel Hill, North Carolina, USA.
ORCID-Profile: Curtis D Collins, Michael P Veve, Ian B Hollis
Zitieren
APA
Collins, C. D., Veve, M. P., & Hollis, I. B. (2026). Rise of the Machines: Comparing Performance of Artificial Intelligence Large Language Models on Pharmacy Specialty Certification Examination Practice Questions. JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY. https://doi.org/10.1002/jac5.70215
Vancouver
Collins CD, Veve MP, Hollis IB. Rise of the Machines: Comparing Performance of Artificial Intelligence Large Language Models on Pharmacy Specialty Certification Examination Practice Questions. JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY. 2026. doi:10.1002/jac5.70215
BibTeX
@article{Collins2026,
author = {Collins, Curtis D and Veve, Michael P and Hollis, Ian B},
title = {Rise of the Machines: Comparing Performance of Artificial Intelligence Large Language Models on Pharmacy Specialty Certification Examination Practice Questions},
journal = {JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY},
year = {2026},
doi = {10.1002/jac5.70215}
}
EUVIMED