EUVIMEDEUVIMED entdecken
VergleichsstudieQuerschnittstudieOpen Access

Natural Language Processing and ICD-10 Coding for Detecting Bleeding Events in Discharge Summaries: Comparative Cross-Sectional Study

Frederic Gaspar iD, Mehdi Zayene iD, Claire Coumau iD, Elliott Bertrand iD, Marie Bettex iD, Marie Annick Le Pogam iD, Chantal Csajka iD

JMIR Medical Informatics · 2025

DOI: 10.2196/67837 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JMIR Medical Informatics, 2025. Verfasst von Frederic Gaspar und 6 weiteren Autor:innen. Beteiligte Einrichtung: Center for Research and Innovation in Clinical Pharmaceutical Sciences, Rue du Bugnon 19, Lausanne, 1011, Switzerland, 41 763306834. Hauptthemen laut MEDLINE: Natural Language Processing, International Classification of Diseases, Hemorrhage, Patient Discharge Summaries. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 60 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Swiss National Science Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Bleeding adverse drug events (ADEs), particularly among older inpatients receiving antithrombotic therapy, represent a major safety concern in hospitals. These events are often underdetected by conventional rule-based systems relying on structured electronic medical record data, such as the ICD-10 (International Statistical Classification of Diseases and Related Health Problems 10th Revision ) codes, which lack the granularity to capture nuanced clinical narratives.

Objective

This study aimed to develop and evaluate a natural language processing (NLP) model to detect and categorize bleeding ADEs in discharge summaries of older adults. Specifically, the model was designed to distinguish between “clinically significant bleeding,” “severe bleeding,” “history of bleeding,” and “no bleeding,” and was compared with a rule-based algorithm using ICD-10 codes.

Methods

Clinicians manually annotated 400 discharge summaries, comprising 65,706 sentences, into four categories: “no bleeding,” “clinically significant bleeding,” “severe bleeding,” and “history of bleeding.” The dataset was divided into a training set (70%, 47,100 sentences) and a test set (30%, 18,606 sentences). Two detection approaches were developed and evaluated: (1) an NLP model using binary logistic regression and support vector machine classifiers, and (2) a traditional rule-based algorithm relying exclusively on predefined ICD-10 codes. To address class imbalance, with most sentences categorized as irrelevant (“no bleeding”), a class-weighting strategy was applied in the NLP model. Model performance was assessed using accuracy, precision, recall, F 1 -score, and receiver operating characteristic (ROC) curve analyses, with manual annotations as the gold standard.

Results

The NLP model significantly outperformed the rule-based approach across all evaluation metrics. At the document level, the NLP model achieved macro-average scores of 0.81 for accuracy and 0.80 for F 1 -score. Precision was particularly high for detecting severe (0.92) and clinically significant bleeding events (0.87), demonstrating strong classification capability despite class imbalance. ROC analyses confirmed the model’s robust diagnostic performance, yielding an area under the curve (AUC) of 0.91 when distinguishing irrelevant sentences from potential bleeding events, 0.88 for identifying historical mentions of bleeding, and notably, 0.94 for differentiating clinically significant from severe bleeding. In contrast, the rule-based ICD-10 model demonstrated high precision (0.94) for clinically significant bleeding but poor recall (0.03) for severe bleeding events, reflecting frequent missed detections. This limitation arose due to its reliance on commonly used ICD-10 codes (eg, gastrointestinal hemorrhage) and inadequate capture of rare severe bleeding conditions such as shock due to hemorrhage.

Conclusions

This study highlights the considerable advantage of NLP over traditional ICD-10 –based methods for detecting bleeding ADEs within electronic medical records. The NLP model effectively captured nuanced clinical narratives, including severity, negations, and historical bleeding events, demonstrating substantial promise for improving patient safety surveillance and clinical decision-making. Future research should extend validation across multiple institutions, diversify annotated datasets, and further refine temporal reasoning capabilities within NLP algorithms.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingInternational Classification of DiseasesHemorrhagePatient Discharge SummariesHumansCross-Sectional StudiesAgedElectronic Health RecordsFemaleMalePatient DischargeAlgorithms

Stichwörter der Autor:innen

AIMLNLPadverse drug eventsartificial intelligencebleedingcross-sectionaldecision support systemsdecision-makingdeep learningdischarge summarieselderlyelectronic medical recordshealthcarehemorrhage

Autor:innen und Institutionen

  • Center for Research and Innovation in Clinical Pharmaceutical Sciences, Rue du Bugnon 19, Lausanne, 1011, Switzerland, 41 763306834.
  • School of Pharmaceutical Sciences, University of Geneva, Geneva, Switzerland.
  • Effixis SA, Lausanne, Switzerland.
  • Institute of Pharmaceutical Sciences of Western Switzerland, University of Geneva, Geneva and Lausanne, Switzerland.
  • Department of Epidemiology and Health Systems, Center for Primary Care and Public Health (Unisanté), University of Lausanne, Lausanne, Switzerland.

ORCID-Profile: Frederic Gaspar, Mehdi Zayene, Claire Coumau, Elliott Bertrand, Marie Bettex, Marie Annick Le Pogam, Chantal Csajka

Förderung

  • Swiss National Science Foundation

Zitieren

APA

Gaspar, F., Zayene, M., Coumau, C., Bertrand, E., Bettex, M., Le Pogam, M. A., & Csajka, C. (2025). Natural Language Processing and ICD-10 Coding for Detecting Bleeding Events in Discharge Summaries: Comparative Cross-Sectional Study. JMIR Medical Informatics. https://doi.org/10.2196/67837

Vancouver

Gaspar F, Zayene M, Coumau C, Bertrand E, Bettex M, Le Pogam MA, et al. Natural Language Processing and ICD-10 Coding for Detecting Bleeding Events in Discharge Summaries: Comparative Cross-Sectional Study. JMIR Medical Informatics. 2025. doi:10.2196/67837

BibTeX
@article{Gaspar2025,
  author = {Gaspar, Frederic and Zayene, Mehdi and Coumau, Claire and Bertrand, Elliott and Bettex, Marie and Le Pogam, Marie Annick and Csajka, Chantal},
  title = {Natural Language Processing and ICD-10 Coding for Detecting Bleeding Events in Discharge Summaries: Comparative Cross-Sectional Study},
  journal = {JMIR Medical Informatics},
  year = {2025},
  doi = {10.2196/67837}
}
Kostenlos starten →