Natural Language Processing and ICD-10 Coding for Detecting Bleeding Events in Discharge Summaries: Comparative Cross-Sectional Study
JMIR Medical Informatics · 2025
DOI: 10.2196/67837 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift JMIR Medical Informatics, 2025. Verfasst von Frederic Gaspar und 6 weiteren Autor:innen. Beteiligte Einrichtung: Center for Research and Innovation in Clinical Pharmaceutical Sciences, Rue du Bugnon 19, Lausanne, 1011, Switzerland, 41 763306834. Hauptthemen laut MEDLINE: Natural Language Processing, International Classification of Diseases, Hemorrhage, Patient Discharge Summaries. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 60 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Swiss National Science Foundation.
Abstract (Originalsprache)
Background
Bleeding adverse drug events (ADEs), particularly among older inpatients receiving antithrombotic therapy, represent a major safety concern in hospitals. These events are often underdetected by conventional rule-based systems relying on structured electronic medical record data, such as the ICD-10 (International Statistical Classification of Diseases and Related Health Problems 10th Revision ) codes, which lack the granularity to capture nuanced clinical narratives.
Objective
This study aimed to develop and evaluate a natural language processing (NLP) model to detect and categorize bleeding ADEs in discharge summaries of older adults. Specifically, the model was designed to distinguish between “clinically significant bleeding,” “severe bleeding,” “history of bleeding,” and “no bleeding,” and was compared with a rule-based algorithm using ICD-10 codes.
Methods
Clinicians manually annotated 400 discharge summaries, comprising 65,706 sentences, into four categories: “no bleeding,” “clinically significant bleeding,” “severe bleeding,” and “history of bleeding.” The dataset was divided into a training set (70%, 47,100 sentences) and a test set (30%, 18,606 sentences). Two detection approaches were developed and evaluated: (1) an NLP model using binary logistic regression and support vector machine classifiers, and (2) a traditional rule-based algorithm relying exclusively on predefined ICD-10 codes. To address class imbalance, with most sentences categorized as irrelevant (“no bleeding”), a class-weighting strategy was applied in the NLP model. Model performance was assessed using accuracy, precision, recall, F 1 -score, and receiver operating characteristic (ROC) curve analyses, with manual annotations as the gold standard.
Results
The NLP model significantly outperformed the rule-based approach across all evaluation metrics. At the document level, the NLP model achieved macro-average scores of 0.81 for accuracy and 0.80 for F 1 -score. Precision was particularly high for detecting severe (0.92) and clinically significant bleeding events (0.87), demonstrating strong classification capability despite class imbalance. ROC analyses confirmed the model’s robust diagnostic performance, yielding an area under the curve (AUC) of 0.91 when distinguishing irrelevant sentences from potential bleeding events, 0.88 for identifying historical mentions of bleeding, and notably, 0.94 for differentiating clinically significant from severe bleeding. In contrast, the rule-based ICD-10 model demonstrated high precision (0.94) for clinically significant bleeding but poor recall (0.03) for severe bleeding events, reflecting frequent missed detections. This limitation arose due to its reliance on commonly used ICD-10 codes (eg, gastrointestinal hemorrhage) and inadequate capture of rare severe bleeding conditions such as shock due to hemorrhage.
Conclusions
This study highlights the considerable advantage of NLP over traditional ICD-10 –based methods for detecting bleeding ADEs within electronic medical records. The NLP model effectively captured nuanced clinical narratives, including severity, negations, and historical bleeding events, demonstrating substantial promise for improving patient safety surveillance and clinical decision-making. Future research should extend validation across multiple institutions, diversify annotated datasets, and further refine temporal reasoning capabilities within NLP algorithms.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Center for Research and Innovation in Clinical Pharmaceutical Sciences, Rue du Bugnon 19, Lausanne, 1011, Switzerland, 41 763306834.
- School of Pharmaceutical Sciences, University of Geneva, Geneva, Switzerland.
- Effixis SA, Lausanne, Switzerland.
- Institute of Pharmaceutical Sciences of Western Switzerland, University of Geneva, Geneva and Lausanne, Switzerland.
- Department of Epidemiology and Health Systems, Center for Primary Care and Public Health (Unisanté), University of Lausanne, Lausanne, Switzerland.
ORCID-Profile: Frederic Gaspar, Mehdi Zayene, Claire Coumau, Elliott Bertrand, Marie Bettex, Marie Annick Le Pogam, Chantal Csajka
Förderung
- Swiss National Science Foundation
Zitieren
APA
Gaspar, F., Zayene, M., Coumau, C., Bertrand, E., Bettex, M., Le Pogam, M. A., & Csajka, C. (2025). Natural Language Processing and ICD-10 Coding for Detecting Bleeding Events in Discharge Summaries: Comparative Cross-Sectional Study. JMIR Medical Informatics. https://doi.org/10.2196/67837
Vancouver
Gaspar F, Zayene M, Coumau C, Bertrand E, Bettex M, Le Pogam MA, et al. Natural Language Processing and ICD-10 Coding for Detecting Bleeding Events in Discharge Summaries: Comparative Cross-Sectional Study. JMIR Medical Informatics. 2025. doi:10.2196/67837
BibTeX
@article{Gaspar2025,
author = {Gaspar, Frederic and Zayene, Mehdi and Coumau, Claire and Bertrand, Elliott and Bettex, Marie and Le Pogam, Marie Annick and Csajka, Chantal},
title = {Natural Language Processing and ICD-10 Coding for Detecting Bleeding Events in Discharge Summaries: Comparative Cross-Sectional Study},
journal = {JMIR Medical Informatics},
year = {2025},
doi = {10.2196/67837}
}
EUVIMED