Is K-fold cross validation the best model selection method for machine learning?
Information Fusion · 2026-11 · 135 · 104404
DOI: 10.1016/j.inffus.2026.104404 ↗Kurz gefasst
Fachartikel in der Zeitschrift Information Fusion, 2026-11. Verfasst von J.M. Gorriz und 5 weiteren Autor:innen. Die Arbeit wurde bisher 25-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Commission, European Regional Development Fund, Spain Ministry of Science Innovation and Universities.
Abstract (Originalsprache)
As a technique that can compactly represent complex patterns, machine learning has significant potential for predictive inference. K-fold cross-validation (CV) is the most common approach to ascertaining the likelihood that a machine learning outcome is generated by chance, and it frequently outperforms conventional hypothesis testing. This improvement uses measures directly obtained from machine learning classifications, such as accuracy, that do not have a parametric description. To approach a frequentist analysis within machine learning pipelines, a permutation test or simple statistics from data partitions (i.e., folds) can be added to estimate confidence intervals. Unfortunately, neither parametric nor non-parametric tests solve the inherent problems of partitioning small sample-size datasets and learning from heterogeneous data sources. The fact that machine learning strongly depends on the learning parameters and the distribution of data across folds recapitulates familiar difficulties around excess false positives and replication. A novel statistical test based on K-fold CV and the Upper Bound of the actual risk (K-fold CUBV) is proposed, where uncertain predictions of machine learning with CV are bounded by the worst case through the evaluation of concentration inequalities. Probably Approximately Correct-Bayesian upper bounds for linear classifiers in combination with K-fold CV are derived and used to estimate the actual risk. The performance with simulated and neuroimaging datasets suggests that K-fold CUBV is a robust criterion for detecting effects and validating accuracy values obtained from machine learning and classical CV schemes, while avoiding excess false positives.
Förderung
- European Commission
- European Regional Development Fund
- Spain Ministry of Science Innovation and Universities
Zitieren
APA
Gorriz, J., Martin-Clemente, R., Segovia, F., Ramirez, J., Ortiz, A., & Suckling, J. (2026). Is K-fold cross validation the best model selection method for machine learning?. Information Fusion, 135, 104404. https://doi.org/10.1016/j.inffus.2026.104404
Vancouver
Gorriz J, Martin-Clemente R, Segovia F, Ramirez J, Ortiz A, Suckling J. Is K-fold cross validation the best model selection method for machine learning?. Information Fusion. 2026;135:104404. doi:10.1016/j.inffus.2026.104404
BibTeX
@article{Gorriz2026,
author = {Gorriz, J.M. and Martin-Clemente, R. and Segovia, F. and Ramirez, J. and Ortiz, A. and Suckling, J.},
title = {Is K-fold cross validation the best model selection method for machine learning?},
journal = {Information Fusion},
year = {2026},
volume = {135},
pages = {104404},
doi = {10.1016/j.inffus.2026.104404}
}
EUVIMED