Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial
JMIR AI · 2023-12-18 · 2 · e49023
DOI: 10.2196/49023 ↗At a glance
Journal article in JMIR AI, published online on 2023-12-18. Written by Drew Wilimitis and 1 co-authors. Institution: Vanderbilt University Medical Center, Vanderbilt University, Nashville, TN, United States. Cited 209 times so far (Crossref); 33 references. The full text is freely available in PubMed Central. Funded by: NHGRI NIH HHS, NIMH NIH HHS, Wellcome Trust.
Abstract (original language)
Cross-validation remains a popular means of developing and validating artificial intelligence for health care. Numerous subtypes of cross-validation exist. Although tutorials on this validation strategy have been published and some with applied examples, we present here a practical tutorial comparing multiple forms of cross-validation using a widely accessible, real-world electronic health care data set: Medical Information Mart for Intensive Care-III (MIMIC-III). This tutorial explored methods such as K-fold cross-validation and nested cross-validation, highlighting their advantages and disadvantages across 2 common predictive modeling use cases: classification (mortality) and regression (length of stay). We aimed to provide readers with reproducible notebooks and best practices for modeling with electronic health care data. We also described sets of useful recommendations as we demonstrated that nested cross-validation reduces optimistic bias but comes with additional computational challenges. This tutorial might improve the community’s understanding of these important methods while catalyzing the modeling community to apply these guides directly in their work using the published code.
Subjects
Author keywords
Authors and institutions
- Vanderbilt University Medical Center, Vanderbilt University, Nashville, TN, United States.
ORCID profiles: Drew Wilimitis, Colin G Walsh
Funding
- NHGRI NIH HHS
- NIMH NIH HHS
- Wellcome Trust
Cite
APA
Wilimitis, D., & Walsh, C. G. (2023). Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial. JMIR AI, 2, e49023. https://doi.org/10.2196/49023
Vancouver
Wilimitis D, Walsh CG. Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial. JMIR AI. 2023;2:e49023. doi:10.2196/49023
BibTeX
@article{Wilimitis2023,
author = {Wilimitis, Drew and Walsh, Colin G},
title = {Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial},
journal = {JMIR AI},
year = {2023},
volume = {2},
pages = {e49023},
doi = {10.2196/49023}
}
EUVIMED