EUVIMEDScopri EUVIMED
Open access

Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial

Drew Wilimitis iD, Colin G Walsh iD

JMIR AI · 2023-12-18 · 2 · e49023

DOI: 10.2196/49023 ↗

At a glance

Journal article in JMIR AI, published online on 2023-12-18. Written by Drew Wilimitis and 1 co-authors. Institution: Vanderbilt University Medical Center, Vanderbilt University, Nashville, TN, United States. Cited 209 times so far (Crossref); 33 references. The full text is freely available in PubMed Central. Funded by: NHGRI NIH HHS, NIMH NIH HHS, Wellcome Trust.

AbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Cross-validation remains a popular means of developing and validating artificial intelligence for health care. Numerous subtypes of cross-validation exist. Although tutorials on this validation strategy have been published and some with applied examples, we present here a practical tutorial comparing multiple forms of cross-validation using a widely accessible, real-world electronic health care data set: Medical Information Mart for Intensive Care-III (MIMIC-III). This tutorial explored methods such as K-fold cross-validation and nested cross-validation, highlighting their advantages and disadvantages across 2 common predictive modeling use cases: classification (mortality) and regression (length of stay). We aimed to provide readers with reproducible notebooks and best practices for modeling with electronic health care data. We also described sets of useful recommendations as we demonstrated that nested cross-validation reduces optimistic bias but comes with additional computational challenges. This tutorial might improve the community’s understanding of these important methods while catalyzing the modeling community to apply these guides directly in their work using the published code.

Subjects

Author keywords

AIartificial intelligenceclinical decision-makingcross-validationdata validationeHealth dataelectronic health carehealth care datamodel developmentpredictive modelingrisk detectiontutorial

Authors and institutions

  • Vanderbilt University Medical Center, Vanderbilt University, Nashville, TN, United States.

ORCID profiles: Drew Wilimitis, Colin G Walsh

Funding

  • NHGRI NIH HHS
  • NIMH NIH HHS
  • Wellcome Trust

Cite

APA

Wilimitis, D., & Walsh, C. G. (2023). Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial. JMIR AI, 2, e49023. https://doi.org/10.2196/49023

Vancouver

Wilimitis D, Walsh CG. Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial. JMIR AI. 2023;2:e49023. doi:10.2196/49023

BibTeX
@article{Wilimitis2023,
  author = {Wilimitis, Drew and Walsh, Colin G},
  title = {Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial},
  journal = {JMIR AI},
  year = {2023},
  volume = {2},
  pages = {e49023},
  doi = {10.2196/49023}
}
Start for free →