Natural Language Processing Identification of Nonprescribed Fentanyl Use in Electronic Health Records: Algorithm Development and Validation Study
Journal of Medical Internet Research · 09.09.2026 · 28 · e97485-e97485
DOI: 10.2196/97485 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 09.09.2026. Verfasst von Joshua Trujeque und 8 weiteren Autor:innen. Beteiligte Einrichtung: Minneapolis Veterans Affairs Healthcare System, Center for Care Delivery and Outcomes Research, One Veterans Drive, Minneapolis, MN, 55417, United St…. Hauptthemen laut MEDLINE: Fentanyl, Natural Language Processing, Electronic Health Records, Algorithms. Die Arbeit nennt 33 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NHLBI NIH HHS, HSRD VA, NIA NIH HHS.
Abstract (Originalsprache)
Background
Overdose and suicide due to nonprescribed fentanyl use have increased significantly, yet health care systems lack reliable methods to identify patients who use nonprescribed fentanyl. International Classification of Diseases codes are inconsistent and do not specify nonprescribed fentanyl use.
Objective
This study aimed to develop natural language processing approaches to identifying nonprescribed fentanyl use in electronic health record (EHR) documentation.
Methods
This retrospective study included Veterans Health Administration patients seen between April 5, 2023, and December 23, 2024. A term list was developed to identify fentanyl-related mentions in clinical text, and 250-character snippets surrounding identified mentions were extracted. Veterans (n=3878) were randomly sampled from 5 predefined groups based on the presence of 1 of 4 terms (“fent,” “blues,” “M30s,” and “tranq”) in their EHR documentation. Physician annotators classified snippets into “nonprescribed fentanyl use,” “prescribed fentanyl use,” or “other,” with interannotator agreement evaluated using the mean pairwise Cohen κ. Cross-validation folds were constructed at the patient level between training and test sets. Penalized logistic regression, Bio-ClinicalBERT, Llama 3-8B, and Mistral-7B were trained on labeled data and compared. Model performance was evaluated using precision, recall, and F 1 -scores for each class, with a focus on the nonprescribed fentanyl use class as the primary label of clinical interest using bootstrapped 95% CIs. A fairness analysis and Shapley additive explanations analysis were performed using Bio-ClinicalBERT. External validation was performed using Bio-ClinicalBERT on an independent sample of 200 snippets, each representing a unique patient from January 2025 to June 2026, with precision reported as the primary validation metric.
Results
Of 7389 snippets, 9.6% (n=709) were classified as “nonprescribed fentanyl use,” 40.3% (n=2981) were classified as “prescribed fentanyl use,” and 50% (n=3699) were classified as “other.” Interannotator agreement was high (κ=0.822). Llama 3-8B achieved the highest F 1 -score for nonprescribed fentanyl use (0.87, 95% CI 0.83-0.92), followed by Mistral-7B (0.80, 95% CI 0.75-0.84), Bio-ClinicalBERT (0.80, 95% CI 0.74-0.85), and penalized logistic regression (0.74, 95% CI 0.73-0.75). Performance was consistent across demographic subgroups, with lower performance for the nonprescribed fentanyl use class observed in female and Hispanic subgroups. Shapley additive explanations analysis revealed clinically meaningful discriminating terms for each class, although subword tokens required contextual interpretation. External validation of Bio-ClinicalBERT demonstrated a precision of 0.79 for nonprescribed fentanyl use.
Conclusions
Natural language processing can identify nonprescribed fentanyl use in EHR documentation, although model performance for this class was lower than overall model performance, reflecting the clinical complexity of identifying nonprescribed use and the variable ways in which clinicians document this problem. This approach may support risk prediction and targeting of interventions to patients exposed to nonprescribed fentanyl.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Substanzen
Autor:innen und Institutionen
- Minneapolis Veterans Affairs Healthcare System, Center for Care Delivery and Outcomes Research, One Veterans Drive, Minneapolis, MN, 55417, United States.
- Rocky Mountain Mental Illness Research, Education, and Clinical Center for Suicide Prevention, Rocky Mountain Regional VA Medical Center, Aurora, CO, United States.
- Injury and Violence Prevention Center, Colorado School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO, United States.
- Center for Veterans Research and Education, Minneapolis, MN, United States.
- Northwell Health, New Hyde Park, NY, United States.
- Department of Psychiatry, Zucker Hillside Hospital, Glen Oaks, NY, United States.
Förderung
- NHLBI NIH HHS
- HSRD VA
- NIA NIH HHS
- NCCIH NIH HHS
- NIDDK NIH HHS
- AHRQ HHS
- NIMHD NIH HHS
- NCI NIH HHS
Zitieren
APA
Trujeque, J., Simonetti, J. A., Ortiz, I., Ingraham, N. E., Mesfin, N., Yeung, J., Zhang, R., Brenner, L. A., & Dudley, R. A. (2026). Natural Language Processing Identification of Nonprescribed Fentanyl Use in Electronic Health Records: Algorithm Development and Validation Study. Journal of Medical Internet Research, 28, e97485-e97485. https://doi.org/10.2196/97485
Vancouver
Trujeque J, Simonetti JA, Ortiz I, Ingraham NE, Mesfin N, Yeung J, et al. Natural Language Processing Identification of Nonprescribed Fentanyl Use in Electronic Health Records: Algorithm Development and Validation Study. Journal of Medical Internet Research. 2026;28:e97485-e97485. doi:10.2196/97485
BibTeX
@article{Trujeque2026,
author = {Trujeque, Joshua and Simonetti, Joseph A and Ortiz, Isai and Ingraham, Nicholas E and Mesfin, Nathan and Yeung, Jeremy and Zhang, Rui and Brenner, Lisa A and Dudley, R Adams},
title = {Natural Language Processing Identification of Nonprescribed Fentanyl Use in Electronic Health Records: Algorithm Development and Validation Study},
journal = {Journal of Medical Internet Research},
year = {2026},
volume = {28},
pages = {e97485--e97485},
doi = {10.2196/97485}
}
EUVIMED