Misleading Results in Posttraumatic Stress Disorder Predictive Models Using Electronic Health Record Data: Algorithm Validation Study
Journal of Medical Internet Research · 2025
DOI: 10.2196/63352 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift Journal of Medical Internet Research, 2025. Verfasst von Thomas M Crow und 5 weiteren Autor:innen. Beteiligte Einrichtung: Behavioral Science Division, National Center for Posttraumatic Stress Disorder, VA Boston Healthcare System, 150 S Huntington Ave, 13B, Boston, MA, 0…. Hauptthemen laut MEDLINE: Stress Disorders, Post-Traumatic, Electronic Health Records, Algorithms. Die Arbeit nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: CSRD VA, NIMH NIH HHS.
Abstract (Originalsprache)
Background
Electronic health record (EHR) data are increasingly used in predictive models of posttraumatic stress disorder (PTSD), but it is unknown how multivariable prediction of an EHR-based diagnosis might differ from prediction of a more rigorous diagnostic criterion. This distinction is important because EHR data are subject to multiple biases, including diagnostic misclassification and differential health care use resulting from factors such as illness severity.
Objective
This study aims to compare predictive models using the same predictors to predict an EHR-based versus semistructured interview-based PTSD diagnostic criterion, quantify model performance discrepancies, and examine potential mechanisms that account for performance differences.
Methods
We compared the performance of several machine learning models predicting EHR-based PTSD diagnosis to models predicting semistructured interview-based diagnosis in a nationwide sample of 1343 US veterans who completed Structured Clinical Interview for DSM-5 ( Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition ) (SCID-5) interviews and had clinic visit data extracted from the Veterans Affairs (VA) EHR. We developed 2 sets of predictive models using 3 algorithms (elastic net regression, random forest, and XGBoost), with a nested cross-validation scheme consisting of an initial train-test split and 10-fold cross-validation within the training set for each type of model. All models used a nearly identical set of predictors including 29 EHR-based visit count variables and 8 demographic variables.
Results
Diagnostic concordance between EHR-based PTSD diagnosis and SCID-5-based PTSD diagnosis was 73.3%, with 17.8% false negatives and 8.9% false positives for EHR-based diagnosis. Models predicting EHR-based PTSD performed very well (area under the receiver operating characteristic curve [AUC] .85-.9; Matthews correlation coefficient [MCC] .58-.69), whereas those predicting interview-based PTSD performed only moderately well overall (AUC .71-.76; MCC .24-.28). Sensitivity analyses showed that participants’ frequency of VA visits played a role in these differences, such that the density of EHR data (proportion of nonzero visit counts across EHR variables) was more associated with EHR-based PTSD diagnosis (b=−0.18, SE 0.02, P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Behavioral Science Division, National Center for Posttraumatic Stress Disorder, VA Boston Healthcare System, 150 S Huntington Ave, 13B, Boston, MA, 02130, United States, 1 6172329500.
- Massachusetts Veterans Epidemiology Research and Information Center (MAVERIC), VA Boston Healthcare System, Boston, MA, United States.
- McLean Institute for Technology in Psychiatry, McLean Hospital, Belmont, MA, United States.
- Department of Psychiatry, Boston University Chobanian & Avedisian School of Medicine, Boston, MA, United States.
ORCID-Profile: Thomas M Crow, Eric Lin, Kelly L Harper, Michael L Crowe, Terence M Keane, Brian P Marx
Förderung
- CSRD VA
- NIMH NIH HHS
Zitieren
APA
Crow, T. M., Lin, E., Harper, K. L., Crowe, M. L., Keane, T. M., & Marx, B. P. (2025). Misleading Results in Posttraumatic Stress Disorder Predictive Models Using Electronic Health Record Data: Algorithm Validation Study. Journal of Medical Internet Research. https://doi.org/10.2196/63352
Vancouver
Crow TM, Lin E, Harper KL, Crowe ML, Keane TM, Marx BP. Misleading Results in Posttraumatic Stress Disorder Predictive Models Using Electronic Health Record Data: Algorithm Validation Study. Journal of Medical Internet Research. 2025. doi:10.2196/63352
BibTeX
@article{Crow2025,
author = {Crow, Thomas M and Lin, Eric and Harper, Kelly L and Crowe, Michael L and Keane, Terence M and Marx, Brian P},
title = {Misleading Results in Posttraumatic Stress Disorder Predictive Models Using Electronic Health Record Data: Algorithm Validation Study},
journal = {Journal of Medical Internet Research},
year = {2025},
doi = {10.2196/63352}
}
EUVIMED