Using Natural Language Processing to Identify Adverse Drug Events Characterized by Medication Replacement in Primary Care Electronic Medical Records: Algorithm and Validation Study
Journal of Medical Internet Research · 10.08.2026 · 28 · e86467-e86467
DOI: 10.2196/86467 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 10.08.2026. Verfasst von Alan Katz und 6 weiteren Autor:innen. Beteiligte Einrichtung: Manitoba Centre for Health Policy, College of Community and Global Health, University of Manitoba, Room 408-727 McDermot Ave, Winnipeg, MB, R3E 3P5, …. Hauptthemen laut MEDLINE: Electronic Health Records, Natural Language Processing, Primary Health Care, Algorithms. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Health care systems generate vast amounts of unstructured text, such as clinical notes, which capture nuanced patient experiences, clinical reasoning, and subtle indicators of health status. While health system research has traditionally relied upon structured data, natural language processing (NLP) enables the extraction of this rich textual information. Leveraging NLP could improve the identification and characterization of underreported adverse drug events (ADEs).
Objective
The primary objective of this study was to train and evaluate multiple NLP models, including both previously published architectures and a novel model, for the identification of ADEs from clinical notes.
Methods
Electronic medical records from the Manitoba Primary Care Research Network (MaPCReN) were used in this study. Clinical notes were annotated to indicate the presence of a possible ADE, the associated words or phrases, and the corresponding drug. A subselection algorithm was applied to ensure sufficient representation of notes containing ADEs for training a robust classifier. The cohort was restricted to patients aged 55 years and older and was annotated in 2 waves: Wave 1 comprised primary care encounter notes selected for a temporally linked emergency department (ED) visit, enriching it for acute presentations, while Wave 2 relaxed this requirement, and its acuity composition was uncharacterized. The annotated data were split into training and test sets. NLP models—including BioBERT, BlueBERT, a large language model (LLM) classifier, and an LLM embeddings–based classifier—were trained on both original clinical notes and notes reformatted into Subjective-Objective-Assessment-Plan (SOAP) structure. To approximate a clinician-inspired reasoning workflow, Mistral-7b-Instruct was used to extract presenting symptoms and generate a ranked list of potential etiologies. Model performance was evaluated using precision, recall, and F 1 -score.
Results
Of the 1085 annotated encounter notes, 355 included ADEs. Across 9 modeling approaches evaluated over 5 random seeds, models trained on SOAP-rewritten notes generally outperformed those trained on original notes. The SOAP Rewrite+ LLM Embeddings Classifier (GritLM-SOAP) achieved the highest mean F 1 -score (72.53%; 95% CI 62.40%‐81.82%), while BioBERT-SOAP weighted achieved the highest mean recall (76.34%; 95% CI 62.69%‐89.66%). End-to-end span-level extraction (named entity recognition+relation extraction) on ADE-positive test notes achieved a mean relaxed F 1 -score of 0.455, with relation extraction identified as the bottleneck.
Conclusions
Effective detection of ADEs in clinical notes may benefit from NLP models that approximate the clinical reasoning of health care providers. While the SOAP Rewrite+ LLM Embeddings Classifier demonstrated a reasonable balance of precision and recall, there is room for improvement as models evolve.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Manitoba Centre for Health Policy, College of Community and Global Health, University of Manitoba, Room 408-727 McDermot Ave, Winnipeg, MB, R3E 3P5, Canada, 1 204 955 2612.
- Department of Family Medicine, Max Rady College of Medicine, University of Manitoba, Winnipeg, MB, Canada.
- College of Community and Global Health, University of Manitoba, Winnipeg, MB, Canada.
- Rady Faculty of Health Sciences, College of Pharmacy, University of Manitoba, Winnipeg, MB, Canada.
Zitieren
APA
Katz, A., Dhankar, A., Fransoo, G., Gordon Pappas, D., Hamad, A. F., Leong, C., & Singer, A. (2026). Using Natural Language Processing to Identify Adverse Drug Events Characterized by Medication Replacement in Primary Care Electronic Medical Records: Algorithm and Validation Study. Journal of Medical Internet Research, 28, e86467-e86467. https://doi.org/10.2196/86467
Vancouver
Katz A, Dhankar A, Fransoo G, Gordon Pappas D, Hamad AF, Leong C, et al. Using Natural Language Processing to Identify Adverse Drug Events Characterized by Medication Replacement in Primary Care Electronic Medical Records: Algorithm and Validation Study. Journal of Medical Internet Research. 2026;28:e86467-e86467. doi:10.2196/86467
BibTeX
@article{Katz2026,
author = {Katz, Alan and Dhankar, Abhishek and Fransoo, Gillian and Gordon Pappas, Diane and Hamad, Amani F and Leong, Christine and Singer, Alexander},
title = {Using Natural Language Processing to Identify Adverse Drug Events Characterized by Medication Replacement in Primary Care Electronic Medical Records: Algorithm and Validation Study},
journal = {Journal of Medical Internet Research},
year = {2026},
volume = {28},
pages = {e86467--e86467},
doi = {10.2196/86467}
}
EUVIMED