EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

Using Natural Language Processing to Identify Adverse Drug Events Characterized by Medication Replacement in Primary Care Electronic Medical Records: Algorithm and Validation Study

Alan Katz, Abhishek Dhankar, Gillian Fransoo, Diane Gordon Pappas, Amani F Hamad, Christine Leong, Alexander Singer

Journal of Medical Internet Research · 10.08.2026 · 28 · e86467-e86467

DOI: 10.2196/86467 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 10.08.2026. Verfasst von Alan Katz und 6 weiteren Autor:innen. Beteiligte Einrichtung: Manitoba Centre for Health Policy, College of Community and Global Health, University of Manitoba, Room 408-727 McDermot Ave, Winnipeg, MB, R3E 3P5, …. Hauptthemen laut MEDLINE: Electronic Health Records, Natural Language Processing, Primary Health Care, Algorithms. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Health care systems generate vast amounts of unstructured text, such as clinical notes, which capture nuanced patient experiences, clinical reasoning, and subtle indicators of health status. While health system research has traditionally relied upon structured data, natural language processing (NLP) enables the extraction of this rich textual information. Leveraging NLP could improve the identification and characterization of underreported adverse drug events (ADEs).

Objective

The primary objective of this study was to train and evaluate multiple NLP models, including both previously published architectures and a novel model, for the identification of ADEs from clinical notes.

Methods

Electronic medical records from the Manitoba Primary Care Research Network (MaPCReN) were used in this study. Clinical notes were annotated to indicate the presence of a possible ADE, the associated words or phrases, and the corresponding drug. A subselection algorithm was applied to ensure sufficient representation of notes containing ADEs for training a robust classifier. The cohort was restricted to patients aged 55 years and older and was annotated in 2 waves: Wave 1 comprised primary care encounter notes selected for a temporally linked emergency department (ED) visit, enriching it for acute presentations, while Wave 2 relaxed this requirement, and its acuity composition was uncharacterized. The annotated data were split into training and test sets. NLP models—including BioBERT, BlueBERT, a large language model (LLM) classifier, and an LLM embeddings–based classifier—were trained on both original clinical notes and notes reformatted into Subjective-Objective-Assessment-Plan (SOAP) structure. To approximate a clinician-inspired reasoning workflow, Mistral-7b-Instruct was used to extract presenting symptoms and generate a ranked list of potential etiologies. Model performance was evaluated using precision, recall, and F 1 -score.

Results

Of the 1085 annotated encounter notes, 355 included ADEs. Across 9 modeling approaches evaluated over 5 random seeds, models trained on SOAP-rewritten notes generally outperformed those trained on original notes. The SOAP Rewrite+ LLM Embeddings Classifier (GritLM-SOAP) achieved the highest mean F 1 -score (72.53%; 95% CI 62.40%‐81.82%), while BioBERT-SOAP weighted achieved the highest mean recall (76.34%; 95% CI 62.69%‐89.66%). End-to-end span-level extraction (named entity recognition+relation extraction) on ADE-positive test notes achieved a mean relaxed F 1 -score of 0.455, with relation extraction identified as the bottleneck.

Conclusions

Effective detection of ADEs in clinical notes may benefit from NLP models that approximate the clinical reasoning of health care providers. While the SOAP Rewrite+ LLM Embeddings Classifier demonstrated a reasonable balance of precision and recall, there is room for improvement as models evolve.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Electronic Health RecordsNatural Language ProcessingPrimary Health CareAlgorithmsDrug-Related Side Effects and Adverse ReactionsHumansMiddle AgedManitoba

Stichwörter der Autor:innen

adverse drug eventselectronic medical recordslarge language modelmachine learningnatural language processing

Autor:innen und Institutionen

  • Manitoba Centre for Health Policy, College of Community and Global Health, University of Manitoba, Room 408-727 McDermot Ave, Winnipeg, MB, R3E 3P5, Canada, 1 204 955 2612.
  • Department of Family Medicine, Max Rady College of Medicine, University of Manitoba, Winnipeg, MB, Canada.
  • College of Community and Global Health, University of Manitoba, Winnipeg, MB, Canada.
  • Rady Faculty of Health Sciences, College of Pharmacy, University of Manitoba, Winnipeg, MB, Canada.

Zitieren

APA

Katz, A., Dhankar, A., Fransoo, G., Gordon Pappas, D., Hamad, A. F., Leong, C., & Singer, A. (2026). Using Natural Language Processing to Identify Adverse Drug Events Characterized by Medication Replacement in Primary Care Electronic Medical Records: Algorithm and Validation Study. Journal of Medical Internet Research, 28, e86467-e86467. https://doi.org/10.2196/86467

Vancouver

Katz A, Dhankar A, Fransoo G, Gordon Pappas D, Hamad AF, Leong C, et al. Using Natural Language Processing to Identify Adverse Drug Events Characterized by Medication Replacement in Primary Care Electronic Medical Records: Algorithm and Validation Study. Journal of Medical Internet Research. 2026;28:e86467-e86467. doi:10.2196/86467

BibTeX
@article{Katz2026,
  author = {Katz, Alan and Dhankar, Abhishek and Fransoo, Gillian and Gordon Pappas, Diane and Hamad, Amani F and Leong, Christine and Singer, Alexander},
  title = {Using Natural Language Processing to Identify Adverse Drug Events Characterized by Medication Replacement in Primary Care Electronic Medical Records: Algorithm and Validation Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e86467--e86467},
  doi = {10.2196/86467}
}
Kostenlos starten →