Evaluation of large language models for antimicrobial classification: implications for antimicrobial stewardship programs
Antimicrobial Stewardship & Healthcare Epidemiology · 2025 · 5 (1)
DOI: 10.1017/ash.2025.10235 ↗Kurz gefasst
Fachartikel in der Zeitschrift Antimicrobial Stewardship & Healthcare Epidemiology, online erschienen am 01.12.2025. Verfasst von Tan Vo und 5 weiteren Autor:innen. Beteiligte Einrichtung: College of Pharmacy, Ferris State University, 220 Ferris Dr., Big Rapids, MI, 49307, USA. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 15 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Objective
To evaluate the ability of large language models (LLMs) with targeted feedback to classify medications as antimicrobial or non-antimicrobial and their implication in antimicrobial stewardship.
Design
Cross-sectional evaluation using a two-phase process: initial unguided classification and feedback-informed reclassification.
Setting
Medication-level analysis of health system prescribing data.
Participants
A data set of 7,239 unique medication entries from health systems in the Collaboration to Harmonize Antimicrobial Registry Measures (CHARM) project.
Methods
Four LLMs, including ChatGPT-3.5, Copilot GPT-4o, Claude Sonnet 4, and Gemini 2.5 Flash, classified all entries against a manual reference standard. Models then received feedback on 20% of misclassified cases for reclassification. Metrics included accuracy, macro F1-score (95% confidence intervals via bootstrap resampling), positive predictive value, negative predictive value, processing time, and error reduction rates (ERRs). McNemar’s test assessed accuracy between phases and model differences.
Results
Baseline accuracy varied among LLMs. Postfeedback, all accuracies improved significantly ( P < .001) to Gemini (99.6%), Claude Sonnet 4 (99.4%), ChatGPT-3.5 (81.0%), and Copilot (79.7%). Gemini achieved the highest macro-F1 (98.9%, 95% CI: 98.4–99.3) and ERR (69.2%). Processing time were fastest for Copilot (42 s), followed by ChatGPT-3.5 (47 s), Gemini (1,080 s), and Claude (3,475 s). Manual classification of this task is estimated to take 18 hours without LLM. Misclassification was most common among antiseptics, antiparasitics, and drugs with antimicrobial components for non-infectious uses (eg, sulfasalazine).
Conclusions
Top-performing LLMs achieved accuracy levels suitable for automating initial antimicrobial classification in stewardship workflows. Performance variability underscores the need for careful selection and continued human oversight in clinical applications. Summary: Four LLMs were evaluated for antimicrobial classification using 7,239 medications. Claude Sonnet 4 and Gemini achieved>99% accuracy, while ChatGPT-3.5 and Copilot showed substantial limitations. Top performers could automate stewardship workflows with appropriate oversight.
Autor:innen und Institutionen
- College of Pharmacy, Ferris State University, 220 Ferris Dr., Big Rapids, MI, 49307, USA.
- Collaborative to Advance Pharmacy Enterprises (CAPE), Grand Rapids, MI, USA.
ORCID-Profile: Tan Vo, Kushal Dahal, Michael Klepser, Benjamin Pontefract, Kaylee E Caniff, Minji Sohn
Zitieren
APA
Vo, T., Dahal, K., Klepser, M., Pontefract, B., Caniff, K. E., & Sohn, M. (2025). Evaluation of large language models for antimicrobial classification: implications for antimicrobial stewardship programs. Antimicrobial Stewardship & Healthcare Epidemiology, 5(1). https://doi.org/10.1017/ash.2025.10235
Vancouver
Vo T, Dahal K, Klepser M, Pontefract B, Caniff KE, Sohn M. Evaluation of large language models for antimicrobial classification: implications for antimicrobial stewardship programs. Antimicrobial Stewardship & Healthcare Epidemiology. 2025;5(1). doi:10.1017/ash.2025.10235
BibTeX
@article{Vo2025,
author = {Vo, Tan and Dahal, Kushal and Klepser, Michael and Pontefract, Benjamin and Caniff, Kaylee E and Sohn, Minji},
title = {Evaluation of large language models for antimicrobial classification: implications for antimicrobial stewardship programs},
journal = {Antimicrobial Stewardship & Healthcare Epidemiology},
year = {2025},
volume = {5},
number = {1},
doi = {10.1017/ash.2025.10235}
}
EUVIMED