EUVIMEDTutustu EUVIMEDiin
Kohortenstudie

Comparison of Rule-Based Natural Language Processing and Diagnosis Code Approaches to Detecting Bleeding in the Electronic Health Record: Algorithm Validation Within a Retrospective Cohort Study

Evan W Draper, Jan A Anderson, Surbhi Shah, Leilia Brown, Margaret R Hanson, Kristin D Negley, Jennifer M Lutz, Andrea E Anderson, John G O'Meara, Nicholas D Jakowenko, Robert D McBane

JMIR Cardio · 2026-11-20 · 10 · e87424-e87424

DOI: 10.2196/87424 ↗

At a glance

Kohortenstudie in JMIR Cardio, published online on 2026-11-20. Written by Evan W Draper and 10 co-authors. 32 references.

TiivistelmäBibliografiset tiedotCiteRelated studies

Abstract (original language)

Background

Anticoagulant use is prevalent and increasing. High-quality anticoagulation management balances individual thrombotic and bleeding risks to optimize care. An automated algorithm to accurately extract major bleeding events from the electronic health record (EHR) of a large health care system would be extremely useful for tracking outcomes of anticoagulated patients.

Objective

The study aims to assess the accuracy of a natural language processing (NLP) approach, compared with and in conjunction with a diagnosis code (Dx Code)–based strategy, for detecting major bleeding from the EHR.

Methods

Patients with at least 1 prescription for apixaban, dabigatran, edoxaban, rivaroxaban, or warfarin between January 1, 2019, and December 31, 2023, were analyzed for major bleeding events. Major bleeding was defined similarly to the International Society on Thrombosis and Haemostasis definition. For the NLP algorithm, clinical notes and radiology reports were screened for mentions of bleeding. MedTagger, an open-source, rule-based NLP tool, was then used on the notes. The NLP results were transformed into a feature set, which was used to create a support vector machine model. Three NLP models were created for central nervous system bleeding, critical site bleeding, and bleeding with a hemoglobin drop or transfusions (other major bleeding). Training and test datasets were created using an active learning approach. The model was trained using 80% of the dataset, while the remaining 20% was used for testing. After model finalization, an independent random sample was used to validate the model. Dx Codes were also used to detect major bleeding, similar to previously published approaches. Five unique algorithms were compared: (1) NLP, (2) Dx Code in the primary position, (3) Dx Code in any position, (4) NLP∩Dx Code in any position, and (5) NLP∪Dx Code in any position. All 5 algorithms were compared with manual chart review.

Results

The final analysis identified 111,338 patients prescribed outpatient anticoagulation during the study. Manual adjudication was performed for 1189 of the 111,338 (1%) patients. Most of the 5 approaches performed well on the test data, with F 1 -scores ranging from 0.545 to 0.959, except for Dx Code–Primary, with an F 1 -score of 0.353 for critical site bleeding. The F 1 -scores for the validation data ranged from 0.509 to 0.679. The NLP∩Dx Code in any position approach yielded excellent precision of 0.864 (95% CI 0.667‐0.953) but only 0.559 (95% CI 0.395‐0.711) sensitivity. The NLP∪Dx Code in any position approach yielded good sensitivity of 0.824 (95% CI 0.665‐0.917) but only 0.368 (95% CI 0.269‐0.481) precision.

Conclusions

Electronic tools can screen the EHRs to identify major bleeding outcomes in large patient cohorts within a multisite health care network with high precision but only moderate sensitivity. Additional work is needed to enhance sensitivity while maintaining high precision.

Cite

APA

Draper, E. W., Anderson, J. A., Shah, S., Brown, L., Hanson, M. R., Negley, K. D., Lutz, J. M., Anderson, A. E., O'Meara, J. G., Jakowenko, N. D., & McBane, R. D. (2026). Comparison of Rule-Based Natural Language Processing and Diagnosis Code Approaches to Detecting Bleeding in the Electronic Health Record: Algorithm Validation Within a Retrospective Cohort Study. JMIR Cardio, 10, e87424-e87424. https://doi.org/10.2196/87424

Vancouver

Draper EW, Anderson JA, Shah S, Brown L, Hanson MR, Negley KD, et al. Comparison of Rule-Based Natural Language Processing and Diagnosis Code Approaches to Detecting Bleeding in the Electronic Health Record: Algorithm Validation Within a Retrospective Cohort Study. JMIR Cardio. 2026;10:e87424-e87424. doi:10.2196/87424

BibTeX
@article{Draper2026,
  author = {Draper, Evan W and Anderson, Jan A and Shah, Surbhi and Brown, Leilia and Hanson, Margaret R and Negley, Kristin D and Lutz, Jennifer M and Anderson, Andrea E and O'Meara, John G and Jakowenko, Nicholas D and McBane, Robert D},
  title = {Comparison of Rule-Based Natural Language Processing and Diagnosis Code Approaches to Detecting Bleeding in the Electronic Health Record: Algorithm Validation Within a Retrospective Cohort Study},
  journal = {JMIR Cardio},
  year = {2026},
  volume = {10},
  pages = {e87424--e87424},
  doi = {10.2196/87424}
}
Start for free →