Reliability of Large Language Model Knowledge Across Brand and Generic Cancer Drug Names
JCO Clinical Cancer Informatics · 2025
DOI: 10.1200/cci-24-00257 ↗Kurz gefasst
Fachartikel in der Zeitschrift JCO Clinical Cancer Informatics, 2025. Verfasst von Jack Gallifant und 8 weiteren Autor:innen. Beteiligte Einrichtung: Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA. Hauptthemen laut MEDLINE: Drugs, Generic, Antineoplastic Agents, Neoplasms, Language. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCI NIH HHS.
Abstract (Originalsprache)
PURPOSE To evaluate the performance and consistency of large language models (LLMs) across brand and generic oncology drug names in various clinical tasks, addressing concerns about potential fluctuations in LLM performance because of subtle phrasing differences that could affect patient care. METHODS This study evaluated three LLMs (GPT-3.5-turbo-0125, GPT-4-turbo, and GPT-4o) using drug names from HemOnc ontology. The assessment included 367 generic-to-brand and 2,516 brand-to-generic pairs, 1,000 drug-drug interaction (DDI) synthetic patient cases, and 2,438 immune-related adverse event (irAE) cases. LLMs were tested on drug name recognition, word association, DDI (DDI) detection, and irAE diagnosis using both brand and generic drug names. RESULTS LLMs demonstrated high accuracy in matching brand and generic names (GPT-4o: 97.38% for brand, 94.71% for generic, P < .01). However, they showed significant inconsistencies in word association tasks. GPT-3.5-turbo-0125 exhibited biases favoring brand names for effectiveness (odds ratio [OR], 1.43, P < .05) and being side-effect-free (OR, 1.76, P < .05). DDI detection accuracy was poor across all models (
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Substanzen
Autor:innen und Institutionen
- Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA.
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA.
- Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA.
- Center for Clinical Cancer Informatics and Data Science, Legorreta Cancer Center, Brown University, Providence, RI.
- Rhode Island Hospital, Providence, RI.
- Universitat Pompeu Fabra, Barcelona, Spain.
ORCID-Profile: Jack Gallifant, Shan Chen, Sandeep K Jain, Umit Topaloglu, Hugo J W L Aerts, Danielle S Bitterman
Förderung
- NCI NIH HHS
Zitieren
APA
Gallifant, J., Chen, S., Jain, S. K., Moreira, P., Topaloglu, U., Aerts, H. J. W. L., Warner, J. L., La Cava, W. G., & Bitterman, D. S. (2025). Reliability of Large Language Model Knowledge Across Brand and Generic Cancer Drug Names. JCO Clinical Cancer Informatics. https://doi.org/10.1200/cci-24-00257
Vancouver
Gallifant J, Chen S, Jain SK, Moreira P, Topaloglu U, Aerts HJWL, et al. Reliability of Large Language Model Knowledge Across Brand and Generic Cancer Drug Names. JCO Clinical Cancer Informatics. 2025. doi:10.1200/cci-24-00257
BibTeX
@article{Gallifant2025,
author = {Gallifant, Jack and Chen, Shan and Jain, Sandeep K and Moreira, Pedro and Topaloglu, Umit and Aerts, Hugo J W L and Warner, Jeremy L and La Cava, William G and Bitterman, Danielle S},
title = {Reliability of Large Language Model Knowledge Across Brand and Generic Cancer Drug Names},
journal = {JCO Clinical Cancer Informatics},
year = {2025},
doi = {10.1200/cci-24-00257}
}
EUVIMED