EUVIMEDObjevte EUVIMED
ValidierungsstudieOpen access

Using Natural Language Processing to Identify Adverse Drug Events Characterized by Medication Replacement in Primary Care Electronic Medical Records: Algorithm and Validation Study

Alan Katz, Abhishek Dhankar, Gillian Fransoo, Diane Gordon Pappas, Amani F Hamad, Christine Leong, Alexander Singer

Journal of Medical Internet Research · 2026-08-10 · 28 · e86467-e86467

DOI: 10.2196/86467 ↗

At a glance

Validierungsstudie in Journal of Medical Internet Research, published online on 2026-08-10. Written by Alan Katz and 6 co-authors. Institution: Manitoba Centre for Health Policy, College of Community and Global Health, University of Manitoba, Room 408-727 McDermot Ave, Winnipeg, MB, R3E 3P5, …. Major MEDLINE topics: Electronic Health Records, Natural Language Processing, Primary Health Care, Algorithms. Cited 1 times so far (Crossref); 39 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Health care systems generate vast amounts of unstructured text, such as clinical notes, which capture nuanced patient experiences, clinical reasoning, and subtle indicators of health status. While health system research has traditionally relied upon structured data, natural language processing (NLP) enables the extraction of this rich textual information. Leveraging NLP could improve the identification and characterization of underreported adverse drug events (ADEs).

Objective

The primary objective of this study was to train and evaluate multiple NLP models, including both previously published architectures and a novel model, for the identification of ADEs from clinical notes.

Methods

Electronic medical records from the Manitoba Primary Care Research Network (MaPCReN) were used in this study. Clinical notes were annotated to indicate the presence of a possible ADE, the associated words or phrases, and the corresponding drug. A subselection algorithm was applied to ensure sufficient representation of notes containing ADEs for training a robust classifier. The cohort was restricted to patients aged 55 years and older and was annotated in 2 waves: Wave 1 comprised primary care encounter notes selected for a temporally linked emergency department (ED) visit, enriching it for acute presentations, while Wave 2 relaxed this requirement, and its acuity composition was uncharacterized. The annotated data were split into training and test sets. NLP models—including BioBERT, BlueBERT, a large language model (LLM) classifier, and an LLM embeddings–based classifier—were trained on both original clinical notes and notes reformatted into Subjective-Objective-Assessment-Plan (SOAP) structure. To approximate a clinician-inspired reasoning workflow, Mistral-7b-Instruct was used to extract presenting symptoms and generate a ranked list of potential etiologies. Model performance was evaluated using precision, recall, and F 1 -score.

Results

Of the 1085 annotated encounter notes, 355 included ADEs. Across 9 modeling approaches evaluated over 5 random seeds, models trained on SOAP-rewritten notes generally outperformed those trained on original notes. The SOAP Rewrite+ LLM Embeddings Classifier (GritLM-SOAP) achieved the highest mean F 1 -score (72.53%; 95% CI 62.40%‐81.82%), while BioBERT-SOAP weighted achieved the highest mean recall (76.34%; 95% CI 62.69%‐89.66%). End-to-end span-level extraction (named entity recognition+relation extraction) on ADE-positive test notes achieved a mean relaxed F 1 -score of 0.455, with relation extraction identified as the bottleneck.

Conclusions

Effective detection of ADEs in clinical notes may benefit from NLP models that approximate the clinical reasoning of health care providers. While the SOAP Rewrite+ LLM Embeddings Classifier demonstrated a reasonable balance of precision and recall, there is room for improvement as models evolve.

Subjects

MeSH terms (MEDLINE)

Electronic Health RecordsNatural Language ProcessingPrimary Health CareAlgorithmsDrug-Related Side Effects and Adverse ReactionsHumansMiddle AgedManitoba

Author keywords

adverse drug eventselectronic medical recordslarge language modelmachine learningnatural language processing

Authors and institutions

  • Manitoba Centre for Health Policy, College of Community and Global Health, University of Manitoba, Room 408-727 McDermot Ave, Winnipeg, MB, R3E 3P5, Canada, 1 204 955 2612.
  • Department of Family Medicine, Max Rady College of Medicine, University of Manitoba, Winnipeg, MB, Canada.
  • College of Community and Global Health, University of Manitoba, Winnipeg, MB, Canada.
  • Rady Faculty of Health Sciences, College of Pharmacy, University of Manitoba, Winnipeg, MB, Canada.

Cite

APA

Katz, A., Dhankar, A., Fransoo, G., Gordon Pappas, D., Hamad, A. F., Leong, C., & Singer, A. (2026). Using Natural Language Processing to Identify Adverse Drug Events Characterized by Medication Replacement in Primary Care Electronic Medical Records: Algorithm and Validation Study. Journal of Medical Internet Research, 28, e86467-e86467. https://doi.org/10.2196/86467

Vancouver

Katz A, Dhankar A, Fransoo G, Gordon Pappas D, Hamad AF, Leong C, et al. Using Natural Language Processing to Identify Adverse Drug Events Characterized by Medication Replacement in Primary Care Electronic Medical Records: Algorithm and Validation Study. Journal of Medical Internet Research. 2026;28:e86467-e86467. doi:10.2196/86467

BibTeX
@article{Katz2026,
  author = {Katz, Alan and Dhankar, Abhishek and Fransoo, Gillian and Gordon Pappas, Diane and Hamad, Amani F and Leong, Christine and Singer, Alexander},
  title = {Using Natural Language Processing to Identify Adverse Drug Events Characterized by Medication Replacement in Primary Care Electronic Medical Records: Algorithm and Validation Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e86467--e86467},
  doi = {10.2196/86467}
}
Start for free →