Towards global model generalizability: independent cross-site feature evaluation for patient-level risk prediction models using the OHDSI network
Journal of the American Medical Informatics Association · 19.04.2024 · 31 (5) · 1051-1061
DOI: 10.1093/jamia/ocae028 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 27.02.2024. Verfasst von Behzad Naderalvojoud und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medicine, Stanford University, Stanford, CA 94305, United States. Hauptthemen laut MEDLINE: Data Science, Medical Informatics. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NLM NIH HHS, NIH HHS, National Library of Medicine.
Abstract (Originalsprache)
Background
Predictive models show promise in healthcare, but their successful deployment is challenging due to limited generalizability. Current external validation often focuses on model performance with restricted feature use from the original training data, lacking insights into their suitability at external sites. Our study introduces an innovative methodology for evaluating features during both the development phase and the validation, focusing on creating and validating predictive models for post-surgery patient outcomes with improved generalizability.
Methods
Electronic health records (EHRs) from 4 countries (United States, United Kingdom, Finland, and Korea) were mapped to the OMOP Common Data Model (CDM), 2008-2019. Machine learning (ML) models were developed to predict post-surgery prolonged opioid use (POU) risks using data collected 6 months before surgery. Both local and cross-site feature selection methods were applied in the development and external validation datasets. Models were developed using Observational Health Data Sciences and Informatics (OHDSI) tools and validated on separate patient cohorts.
Results
Model development included 41 929 patients, 14.6% with POU. The external validation included 31 932 (UK), 23 100 (US), 7295 (Korea), and 3934 (Finland) patients with POU of 44.2%, 22.0%, 15.8%, and 21.8%, respectively. The top-performing model, Lasso logistic regression, achieved an area under the receiver operating characteristic curve (AUROC) of 0.75 during local validation and 0.69 (SD = 0.02) (averaged) in external validation. Models trained with cross-site feature selection significantly outperformed those using only features from the development site through external validation (P < .05).
Conclusions
Using EHRs across four countries mapped to the OMOP CDM, we developed generalizable predictive models for POU. Our approach demonstrates the significant impact of cross-site feature selection in improving model performance, underscoring the importance of incorporating diverse feature sets from various clinical settings to enhance the generalizability and utility of predictive healthcare models.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Medicine, Stanford University, Stanford, CA 94305, United States.
- Department of Surgery, Veterans Affairs Palo Alto Health Care System, Palo Alto, CA 94304, United States.
- KI Research Institute, Kfar Malal, 4592000, Israel.
- Department of Biomedical Informatics, Ajou University Graduate School of Medicine, Suwon, 16499, Korea.
- Institute for Molecular Medicine Finland (FIMM), HiLIFE, University of Helsinki, Helsinki, 00014, Finland.
- Department of Biomedical Informatics, Columbia University, New York, NY 10032, United States.
ORCID-Profile: Behzad Naderalvojoud, Tina Hernandez-Boussard
Förderung
- NLM NIH HHS
- NIH HHS
- National Library of Medicine
- National Institutes of Health
Zitieren
APA
Naderalvojoud, B., Curtin, C. M., Yanover, C., El-Hay, T., Choi, B., Park, R. W., Tabuenca, J. G., Reeve, M. P., Falconer, T., Humphreys, K., Asch, S. M., & Hernandez-Boussard, T. (2024). Towards global model generalizability: independent cross-site feature evaluation for patient-level risk prediction models using the OHDSI network. Journal of the American Medical Informatics Association, 31(5), 1051-1061. https://doi.org/10.1093/jamia/ocae028
Vancouver
Naderalvojoud B, Curtin CM, Yanover C, El-Hay T, Choi B, Park RW, et al. Towards global model generalizability: independent cross-site feature evaluation for patient-level risk prediction models using the OHDSI network. Journal of the American Medical Informatics Association. 2024;31(5):1051-1061. doi:10.1093/jamia/ocae028
BibTeX
@article{Naderalvojoud2024,
author = {Naderalvojoud, Behzad and Curtin, Catherine M and Yanover, Chen and El-Hay, Tal and Choi, Byungjin and Park, Rae Woong and Tabuenca, Javier Gracia and Reeve, Mary Pat and Falconer, Thomas and Humphreys, Keith and Asch, Steven M and Hernandez-Boussard, Tina},
title = {Towards global model generalizability: independent cross-site feature evaluation for patient-level risk prediction models using the OHDSI network},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
volume = {31},
number = {5},
pages = {1051--1061},
doi = {10.1093/jamia/ocae028}
}
EUVIMED