EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluation of large language models for antimicrobial classification: implications for antimicrobial stewardship programs

Tan Vo iD, Kushal Dahal iD, Michael Klepser iD, Benjamin Pontefract iD, Kaylee E Caniff iD, Minji Sohn iD

Antimicrobial Stewardship & Healthcare Epidemiology · 2025 · 5 (1)

DOI: 10.1017/ash.2025.10235 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Antimicrobial Stewardship & Healthcare Epidemiology, online erschienen am 01.12.2025. Verfasst von Tan Vo und 5 weiteren Autor:innen. Beteiligte Einrichtung: College of Pharmacy, Ferris State University, 220 Ferris Dr., Big Rapids, MI, 49307, USA. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 15 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To evaluate the ability of large language models (LLMs) with targeted feedback to classify medications as antimicrobial or non-antimicrobial and their implication in antimicrobial stewardship.

Design

Cross-sectional evaluation using a two-phase process: initial unguided classification and feedback-informed reclassification.

Setting

Medication-level analysis of health system prescribing data.

Participants

A data set of 7,239 unique medication entries from health systems in the Collaboration to Harmonize Antimicrobial Registry Measures (CHARM) project.

Methods

Four LLMs, including ChatGPT-3.5, Copilot GPT-4o, Claude Sonnet 4, and Gemini 2.5 Flash, classified all entries against a manual reference standard. Models then received feedback on 20% of misclassified cases for reclassification. Metrics included accuracy, macro F1-score (95% confidence intervals via bootstrap resampling), positive predictive value, negative predictive value, processing time, and error reduction rates (ERRs). McNemar’s test assessed accuracy between phases and model differences.

Results

Baseline accuracy varied among LLMs. Postfeedback, all accuracies improved significantly ( P < .001) to Gemini (99.6%), Claude Sonnet 4 (99.4%), ChatGPT-3.5 (81.0%), and Copilot (79.7%). Gemini achieved the highest macro-F1 (98.9%, 95% CI: 98.4–99.3) and ERR (69.2%). Processing time were fastest for Copilot (42 s), followed by ChatGPT-3.5 (47 s), Gemini (1,080 s), and Claude (3,475 s). Manual classification of this task is estimated to take 18 hours without LLM. Misclassification was most common among antiseptics, antiparasitics, and drugs with antimicrobial components for non-infectious uses (eg, sulfasalazine).

Conclusions

Top-performing LLMs achieved accuracy levels suitable for automating initial antimicrobial classification in stewardship workflows. Performance variability underscores the need for careful selection and continued human oversight in clinical applications. Summary: Four LLMs were evaluated for antimicrobial classification using 7,239 medications. Claude Sonnet 4 and Gemini achieved>99% accuracy, while ChatGPT-3.5 and Copilot showed substantial limitations. Top performers could automate stewardship workflows with appropriate oversight.

Autor:innen und Institutionen

  • College of Pharmacy, Ferris State University, 220 Ferris Dr., Big Rapids, MI, 49307, USA.
  • Collaborative to Advance Pharmacy Enterprises (CAPE), Grand Rapids, MI, USA.

ORCID-Profile: Tan Vo, Kushal Dahal, Michael Klepser, Benjamin Pontefract, Kaylee E Caniff, Minji Sohn

Zitieren

APA

Vo, T., Dahal, K., Klepser, M., Pontefract, B., Caniff, K. E., & Sohn, M. (2025). Evaluation of large language models for antimicrobial classification: implications for antimicrobial stewardship programs. Antimicrobial Stewardship & Healthcare Epidemiology, 5(1). https://doi.org/10.1017/ash.2025.10235

Vancouver

Vo T, Dahal K, Klepser M, Pontefract B, Caniff KE, Sohn M. Evaluation of large language models for antimicrobial classification: implications for antimicrobial stewardship programs. Antimicrobial Stewardship & Healthcare Epidemiology. 2025;5(1). doi:10.1017/ash.2025.10235

BibTeX
@article{Vo2025,
  author = {Vo, Tan and Dahal, Kushal and Klepser, Michael and Pontefract, Benjamin and Caniff, Kaylee E and Sohn, Minji},
  title = {Evaluation of large language models for antimicrobial classification: implications for antimicrobial stewardship programs},
  journal = {Antimicrobial Stewardship & Healthcare Epidemiology},
  year = {2025},
  volume = {5},
  number = {1},
  doi = {10.1017/ash.2025.10235}
}
Kostenlos starten →