Biomedical Large Language Models and Prompt Engineering for Causality Assessment of Individual Case Safety Reports in Pharmacovigilance
Pharmaceutical Research · 2026-08 · 43 (8) · 2761-2774
DOI: 10.1007/s11095-026-04121-4 ↗Kurz gefasst
Fachartikel in der Zeitschrift Pharmaceutical Research, online erschienen am 23.05.2026. Verfasst von Nicole Sonne Heckmann und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Drug Design and Pharmacology, University of Copenhagen, Copenhagen, Denmark. Hauptthemen laut MEDLINE: Large Language Models, Pharmacovigilance, Adverse Drug Reaction Reporting Systems, Drug-Related Side Effects and Adverse Reactions. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Copenhagen University.
Abstract (Originalsprache)
Background
Biomedical Large Language Models (LLMs) combined with prompt engineering offer domain-specific reasoning, yet their application to individual-level causality assessment remains unexplored. This study evaluated five combinations of biomedical LLMs, prompting strategies, and causality algorithms by comparing their agreement with two human expert evaluators. Research Design and Methods. A total of 150 Individual Case Safety Reports (ICSRs) were analyzed: 140 reports from Food and Drug Administration Adverse Event Reporting System (FAERS), and 10 myocarditis/pericarditis ICSRs from Vaccine AERS (VAERS). Assessments were conducted using the Naranjo and WHO-UMC algorithms. Biomedical LLMs tested included TinyLlama 1.1B, Medicine LLaMA-3 8B, and MedLLaMA v20, combined with Chain-of-Thought (CoT) or Decomposition prompting. Agreement was measured using Gwet’s Agreement Coefficient 1 (AC1) and percentage agreement, alongside performance metrics and qualitative error analysis.
Results
The Medicine LLaMA-3 8B-Naranjo-CoT combination achieved the highest agreement with human assessors for the final classification of causality (64%). Biomedical LLMs demonstrated low inter-rater agreement on critical items of causality assessment such as identification of listed AE, temporal plausibility, alternative causes, and objective evidence of AEs. Frequent model failures included irrelevant responses.
Conclusions
Biomedical LLMs showed improved performance over general purpose models previously tested but remain suboptimal for reliable causality assessment of ICSRs.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Drug Design and Pharmacology, University of Copenhagen, Copenhagen, Denmark.
- Department of Clinical and Experimental Medicine, University of Messina, Messina, Italy.
- Department of Biomedical and Clinical Sciences (DIBIC), Università Degli Studi Di Milano, Milan, Italy.
- Safety Operations, Novo Nordisk, Søborg, Denmark.
- Department of Drug Design and Pharmacology, University of Copenhagen, Copenhagen, Denmark. maurizio.sessa@sund.ku.dk.
Förderung
- Copenhagen University
Zitieren
APA
Heckmann, N. S., Papoutsi, D. G., Barbieri, M. A., Battini, V., Mølgaard, S. N., Schmidt, S. Ø., Melskens, L., & Sessa, M. (2026). Biomedical Large Language Models and Prompt Engineering for Causality Assessment of Individual Case Safety Reports in Pharmacovigilance. Pharmaceutical Research, 43(8), 2761-2774. https://doi.org/10.1007/s11095-026-04121-4
Vancouver
Heckmann NS, Papoutsi DG, Barbieri MA, Battini V, Mølgaard SN, Schmidt SØ, et al. Biomedical Large Language Models and Prompt Engineering for Causality Assessment of Individual Case Safety Reports in Pharmacovigilance. Pharmaceutical Research. 2026;43(8):2761-2774. doi:10.1007/s11095-026-04121-4
BibTeX
@article{Heckmann2026,
author = {Heckmann, Nicole Sonne and Papoutsi, Despoina Georgia and Barbieri, Maria Antonietta and Battini, Vera and Mølgaard, Søren Norlin and Schmidt, Simon Ørum and Melskens, Lars and Sessa, Maurizio},
title = {Biomedical Large Language Models and Prompt Engineering for Causality Assessment of Individual Case Safety Reports in Pharmacovigilance},
journal = {Pharmaceutical Research},
year = {2026},
volume = {43},
number = {8},
pages = {2761--2774},
doi = {10.1007/s11095-026-04121-4}
}
EUVIMED