Comparison of Rule-Based Natural Language Processing and Diagnosis Code Approaches to Detecting Bleeding in the Electronic Health Record: Algorithm Validation Within a Retrospective Cohort Study
JMIR Cardio · 2026-11-20 · 10 · e87424-e87424
DOI: 10.2196/87424 ↗At a glance
Kohortenstudie in JMIR Cardio, published online on 2026-11-20. Written by Evan W Draper and 10 co-authors. 32 references.
Abstract (original language)
Background
Anticoagulant use is prevalent and increasing. High-quality anticoagulation management balances individual thrombotic and bleeding risks to optimize care. An automated algorithm to accurately extract major bleeding events from the electronic health record (EHR) of a large health care system would be extremely useful for tracking outcomes of anticoagulated patients.
Objective
The study aims to assess the accuracy of a natural language processing (NLP) approach, compared with and in conjunction with a diagnosis code (Dx Code)–based strategy, for detecting major bleeding from the EHR.
Methods
Patients with at least 1 prescription for apixaban, dabigatran, edoxaban, rivaroxaban, or warfarin between January 1, 2019, and December 31, 2023, were analyzed for major bleeding events. Major bleeding was defined similarly to the International Society on Thrombosis and Haemostasis definition. For the NLP algorithm, clinical notes and radiology reports were screened for mentions of bleeding. MedTagger, an open-source, rule-based NLP tool, was then used on the notes. The NLP results were transformed into a feature set, which was used to create a support vector machine model. Three NLP models were created for central nervous system bleeding, critical site bleeding, and bleeding with a hemoglobin drop or transfusions (other major bleeding). Training and test datasets were created using an active learning approach. The model was trained using 80% of the dataset, while the remaining 20% was used for testing. After model finalization, an independent random sample was used to validate the model. Dx Codes were also used to detect major bleeding, similar to previously published approaches. Five unique algorithms were compared: (1) NLP, (2) Dx Code in the primary position, (3) Dx Code in any position, (4) NLP∩Dx Code in any position, and (5) NLP∪Dx Code in any position. All 5 algorithms were compared with manual chart review.
Results
The final analysis identified 111,338 patients prescribed outpatient anticoagulation during the study. Manual adjudication was performed for 1189 of the 111,338 (1%) patients. Most of the 5 approaches performed well on the test data, with F 1 -scores ranging from 0.545 to 0.959, except for Dx Code–Primary, with an F 1 -score of 0.353 for critical site bleeding. The F 1 -scores for the validation data ranged from 0.509 to 0.679. The NLP∩Dx Code in any position approach yielded excellent precision of 0.864 (95% CI 0.667‐0.953) but only 0.559 (95% CI 0.395‐0.711) sensitivity. The NLP∪Dx Code in any position approach yielded good sensitivity of 0.824 (95% CI 0.665‐0.917) but only 0.368 (95% CI 0.269‐0.481) precision.
Conclusions
Electronic tools can screen the EHRs to identify major bleeding outcomes in large patient cohorts within a multisite health care network with high precision but only moderate sensitivity. Additional work is needed to enhance sensitivity while maintaining high precision.
Cite
APA
Draper, E. W., Anderson, J. A., Shah, S., Brown, L., Hanson, M. R., Negley, K. D., Lutz, J. M., Anderson, A. E., O'Meara, J. G., Jakowenko, N. D., & McBane, R. D. (2026). Comparison of Rule-Based Natural Language Processing and Diagnosis Code Approaches to Detecting Bleeding in the Electronic Health Record: Algorithm Validation Within a Retrospective Cohort Study. JMIR Cardio, 10, e87424-e87424. https://doi.org/10.2196/87424
Vancouver
Draper EW, Anderson JA, Shah S, Brown L, Hanson MR, Negley KD, et al. Comparison of Rule-Based Natural Language Processing and Diagnosis Code Approaches to Detecting Bleeding in the Electronic Health Record: Algorithm Validation Within a Retrospective Cohort Study. JMIR Cardio. 2026;10:e87424-e87424. doi:10.2196/87424
BibTeX
@article{Draper2026,
author = {Draper, Evan W and Anderson, Jan A and Shah, Surbhi and Brown, Leilia and Hanson, Margaret R and Negley, Kristin D and Lutz, Jennifer M and Anderson, Andrea E and O'Meara, John G and Jakowenko, Nicholas D and McBane, Robert D},
title = {Comparison of Rule-Based Natural Language Processing and Diagnosis Code Approaches to Detecting Bleeding in the Electronic Health Record: Algorithm Validation Within a Retrospective Cohort Study},
journal = {JMIR Cardio},
year = {2026},
volume = {10},
pages = {e87424--e87424},
doi = {10.2196/87424}
}
EUVIMED