EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Biomedical Large Language Models and Prompt Engineering for Causality Assessment of Individual Case Safety Reports in Pharmacovigilance

Nicole Sonne Heckmann, Despoina Georgia Papoutsi, Maria Antonietta Barbieri, Vera Battini, Søren Norlin Mølgaard, Simon Ørum Schmidt, Lars Melskens, Maurizio Sessa

Pharmaceutical Research · 2026-08 · 43 (8) · 2761-2774

DOI: 10.1007/s11095-026-04121-4 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Pharmaceutical Research, online erschienen am 23.05.2026. Verfasst von Nicole Sonne Heckmann und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Drug Design and Pharmacology, University of Copenhagen, Copenhagen, Denmark. Hauptthemen laut MEDLINE: Large Language Models, Pharmacovigilance, Adverse Drug Reaction Reporting Systems, Drug-Related Side Effects and Adverse Reactions. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Copenhagen University.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Biomedical Large Language Models (LLMs) combined with prompt engineering offer domain-specific reasoning, yet their application to individual-level causality assessment remains unexplored. This study evaluated five combinations of biomedical LLMs, prompting strategies, and causality algorithms by comparing their agreement with two human expert evaluators. Research Design and Methods. A total of 150 Individual Case Safety Reports (ICSRs) were analyzed: 140 reports from Food and Drug Administration Adverse Event Reporting System (FAERS), and 10 myocarditis/pericarditis ICSRs from Vaccine AERS (VAERS). Assessments were conducted using the Naranjo and WHO-UMC algorithms. Biomedical LLMs tested included TinyLlama 1.1B, Medicine LLaMA-3 8B, and MedLLaMA v20, combined with Chain-of-Thought (CoT) or Decomposition prompting. Agreement was measured using Gwet’s Agreement Coefficient 1 (AC1) and percentage agreement, alongside performance metrics and qualitative error analysis.

Results

The Medicine LLaMA-3 8B-Naranjo-CoT combination achieved the highest agreement with human assessors for the final classification of causality (64%). Biomedical LLMs demonstrated low inter-rater agreement on critical items of causality assessment such as identification of listed AE, temporal plausibility, alternative causes, and objective evidence of AEs. Frequent model failures included irrelevant responses.

Conclusions

Biomedical LLMs showed improved performance over general purpose models previously tested but remain suboptimal for reliable causality assessment of ICSRs.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsPharmacovigilanceAdverse Drug Reaction Reporting SystemsDrug-Related Side Effects and Adverse ReactionsHumansAlgorithmsCausalityUnited States

Stichwörter der Autor:innen

Artificial intelligenceBiomedical large language modelCausality assessmentSpontaneous reporting system database

Autor:innen und Institutionen

  • Department of Drug Design and Pharmacology, University of Copenhagen, Copenhagen, Denmark.
  • Department of Clinical and Experimental Medicine, University of Messina, Messina, Italy.
  • Department of Biomedical and Clinical Sciences (DIBIC), Università Degli Studi Di Milano, Milan, Italy.
  • Safety Operations, Novo Nordisk, Søborg, Denmark.
  • Department of Drug Design and Pharmacology, University of Copenhagen, Copenhagen, Denmark. maurizio.sessa@sund.ku.dk.

Förderung

  • Copenhagen University

Zitieren

APA

Heckmann, N. S., Papoutsi, D. G., Barbieri, M. A., Battini, V., Mølgaard, S. N., Schmidt, S. Ø., Melskens, L., & Sessa, M. (2026). Biomedical Large Language Models and Prompt Engineering for Causality Assessment of Individual Case Safety Reports in Pharmacovigilance. Pharmaceutical Research, 43(8), 2761-2774. https://doi.org/10.1007/s11095-026-04121-4

Vancouver

Heckmann NS, Papoutsi DG, Barbieri MA, Battini V, Mølgaard SN, Schmidt SØ, et al. Biomedical Large Language Models and Prompt Engineering for Causality Assessment of Individual Case Safety Reports in Pharmacovigilance. Pharmaceutical Research. 2026;43(8):2761-2774. doi:10.1007/s11095-026-04121-4

BibTeX
@article{Heckmann2026,
  author = {Heckmann, Nicole Sonne and Papoutsi, Despoina Georgia and Barbieri, Maria Antonietta and Battini, Vera and Mølgaard, Søren Norlin and Schmidt, Simon Ørum and Melskens, Lars and Sessa, Maurizio},
  title = {Biomedical Large Language Models and Prompt Engineering for Causality Assessment of Individual Case Safety Reports in Pharmacovigilance},
  journal = {Pharmaceutical Research},
  year = {2026},
  volume = {43},
  number = {8},
  pages = {2761--2774},
  doi = {10.1007/s11095-026-04121-4}
}
Kostenlos starten →