EUVIMEDEUVIMED entdecken
Open Access

Reliability of Large Language Model Knowledge Across Brand and Generic Cancer Drug Names

Jack Gallifant iD, Shan Chen iD, Sandeep K Jain iD, Pedro Moreira, Umit Topaloglu iD, Hugo J W L Aerts iD, Jeremy L Warner, William G La Cava, Danielle S Bitterman iD

JCO Clinical Cancer Informatics · 2025

DOI: 10.1200/cci-24-00257 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JCO Clinical Cancer Informatics, 2025. Verfasst von Jack Gallifant und 8 weiteren Autor:innen. Beteiligte Einrichtung: Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA. Hauptthemen laut MEDLINE: Drugs, Generic, Antineoplastic Agents, Neoplasms, Language. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCI NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

PURPOSE To evaluate the performance and consistency of large language models (LLMs) across brand and generic oncology drug names in various clinical tasks, addressing concerns about potential fluctuations in LLM performance because of subtle phrasing differences that could affect patient care. METHODS This study evaluated three LLMs (GPT-3.5-turbo-0125, GPT-4-turbo, and GPT-4o) using drug names from HemOnc ontology. The assessment included 367 generic-to-brand and 2,516 brand-to-generic pairs, 1,000 drug-drug interaction (DDI) synthetic patient cases, and 2,438 immune-related adverse event (irAE) cases. LLMs were tested on drug name recognition, word association, DDI (DDI) detection, and irAE diagnosis using both brand and generic drug names. RESULTS LLMs demonstrated high accuracy in matching brand and generic names (GPT-4o: 97.38% for brand, 94.71% for generic, P < .01). However, they showed significant inconsistencies in word association tasks. GPT-3.5-turbo-0125 exhibited biases favoring brand names for effectiveness (odds ratio [OR], 1.43, P < .05) and being side-effect-free (OR, 1.76, P < .05). DDI detection accuracy was poor across all models (

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Drugs, GenericAntineoplastic AgentsNeoplasmsLanguageTerminology as TopicHumansDrug InteractionsReproducibility of ResultsLarge Language Models

Substanzen

Drugs, GenericAntineoplastic Agents

Autor:innen und Institutionen

  • Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA.
  • Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA.
  • Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA.
  • Center for Clinical Cancer Informatics and Data Science, Legorreta Cancer Center, Brown University, Providence, RI.
  • Rhode Island Hospital, Providence, RI.
  • Universitat Pompeu Fabra, Barcelona, Spain.

ORCID-Profile: Jack Gallifant, Shan Chen, Sandeep K Jain, Umit Topaloglu, Hugo J W L Aerts, Danielle S Bitterman

Förderung

  • NCI NIH HHS

Zitieren

APA

Gallifant, J., Chen, S., Jain, S. K., Moreira, P., Topaloglu, U., Aerts, H. J. W. L., Warner, J. L., La Cava, W. G., & Bitterman, D. S. (2025). Reliability of Large Language Model Knowledge Across Brand and Generic Cancer Drug Names. JCO Clinical Cancer Informatics. https://doi.org/10.1200/cci-24-00257

Vancouver

Gallifant J, Chen S, Jain SK, Moreira P, Topaloglu U, Aerts HJWL, et al. Reliability of Large Language Model Knowledge Across Brand and Generic Cancer Drug Names. JCO Clinical Cancer Informatics. 2025. doi:10.1200/cci-24-00257

BibTeX
@article{Gallifant2025,
  author = {Gallifant, Jack and Chen, Shan and Jain, Sandeep K and Moreira, Pedro and Topaloglu, Umit and Aerts, Hugo J W L and Warner, Jeremy L and La Cava, William G and Bitterman, Danielle S},
  title = {Reliability of Large Language Model Knowledge Across Brand and Generic Cancer Drug Names},
  journal = {JCO Clinical Cancer Informatics},
  year = {2025},
  doi = {10.1200/cci-24-00257}
}
Kostenlos starten →