EUVIMEDOpdag EUVIMED
Crossref

Natural Language Processing Supported by Annotation Process Artifacts to Detect Risky Alcohol Use in Electronic Health Records: Algorithm Development and Validation Study

Katherine Weber, Asher Strayhorn, Xintong Ju, Jessica Mellinger, Gerald Scott Winder, Anne C Fernandez, VG Vinod Vydiswaran

Journal of Medical Internet Research · 2026-11-20 · 28 · e91449

DOI: 10.2196/91449 ↗

At a glance

Journal article in Journal of Medical Internet Research, published online on 2026-11-20. Written by Katherine Weber and 6 co-authors. 48 references.

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

High-risk alcohol use and alcohol use disorder (AUD) are significant contributors to surgical risk and complications yet remain underidentified in surgical populations. While electronic health records (EHRs) contain relevant information about patients’ alcohol use in clinical notes, manual chart review is time-intensive and impractical at scale. Natural language processing (NLP) offers a path to automated identification, but alcohol-related risk classes are highly imbalanced in clinical populations, making multiclass classification particularly challenging.

Objective

This study aims to develop an automated NLP pipeline for 4-class alcohol risk classification using clinical notes in a surgical population, addressing class imbalance by leveraging intermediate annotation evidence generated during the human labeling process.

Methods

We manually labeled 1000 surgical patients drawn from a large academic health system into ordinal alcohol risk classes (no-alcohol, low-risk, high-risk, and probable-dependence) and a fifth class indicating no mention of alcohol use. We used a structured annotation process with 6 trained annotators and expert-developed guidelines. We developed a multistage NLP pipeline operating on 3 years of clinical notes per patient. First, a rules-based preprocessing stage filtered notes to clinically relevant sections containing alcohol-related content. No-mention cases were excluded from further modeling. Second, a BioClinicalBERT-based multilabel classifier (“auto-annotator”) was fine-tuned to reproduce the annotations created by human labelers at the section level, producing a structured 26-dimensional probability vector for each section. Finally, a bidirectional long short-term memory (LSTM) network classified patients into the 4 risk categories by processing each patient’s ordered sequence of auto-annotation vectors, allowing the model to capture temporally accumulating evidence across clinical encounters.

Results

Human annotators achieved 93.7% first-pass agreement overall (Fleiss κ=0.8891), but agreement was as low as 67.5% for probable-dependence cases. The 4-class LSTM classifier achieved a weighted-average F1-score of 0.86 and a macroaverage F1-score of 0.77 on a held-out validation set of 195 patients. One-vs-rest area under the curve (AUC) scores were 0.97 for no-alcohol, 0.88 for low-risk, 0.90 for probable-dependence, and 0.71 for high-risk. Collapsed to a binary high/low risk classification, the model achieved an accuracy of 0.92 and a macro-F1-score of 0.80. Performance was comparable across biological sex without explicit sex-specific training.

Conclusions

We used a 2-stage NLP pipeline to identify 4 ordinal classes of alcohol use risk in surgical patients. Four-class alcohol-use classification presents a highly imbalanced problem that constrains the availability of high-importance but rare labels in the dataset. We addressed this data scarcity by using intermediate annotation evidence as well as final labels to enrich the feature space for the classification task. The pipeline was able to separate patient records into 4 strata of alcohol-use risk, with “high-risk” patients being most likely to be misclassified into an adjacent class.

Cite

APA

Weber, K., Strayhorn, A., Ju, X., Mellinger, J., Winder, G. S., Fernandez, A. C., & Vydiswaran, V. V. (2026). Natural Language Processing Supported by Annotation Process Artifacts to Detect Risky Alcohol Use in Electronic Health Records: Algorithm Development and Validation Study. Journal of Medical Internet Research, 28, e91449. https://doi.org/10.2196/91449

Vancouver

Weber K, Strayhorn A, Ju X, Mellinger J, Winder GS, Fernandez AC, et al. Natural Language Processing Supported by Annotation Process Artifacts to Detect Risky Alcohol Use in Electronic Health Records: Algorithm Development and Validation Study. Journal of Medical Internet Research. 2026;28:e91449. doi:10.2196/91449

BibTeX
@article{Weber2026,
  author = {Weber, Katherine and Strayhorn, Asher and Ju, Xintong and Mellinger, Jessica and Winder, Gerald Scott and Fernandez, Anne C and Vydiswaran, VG Vinod},
  title = {Natural Language Processing Supported by Annotation Process Artifacts to Detect Risky Alcohol Use in Electronic Health Records: Algorithm Development and Validation Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e91449},
  doi = {10.2196/91449}
}
Start for free →