Leakage and the reproducibility crisis in machine-learning-based science
Patterns · 2023-09 · 4 (9) · 100804
DOI: 10.1016/j.patter.2023.100804 ↗Kurz gefasst
Fachartikel in der Zeitschrift Patterns, 2023-09. Verfasst von Sayash Kapoor und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Center for Information Technology Policy, Princeton University, Princeton, NJ 08540, USA. Die Arbeit wurde bisher 1560-mal zitiert (Crossref) und nennt 104 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation.
Abstract (Originalsprache)
Machine-learning (ML) methods have gained prominence in the quantitative sciences. However, there are many known methodological pitfalls, including data leakage, in ML-based science. We systematically investigate reproducibility issues in ML-based science. Through a survey of literature in fields that have adopted ML methods, we find 17 fields where leakage has been found, collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions. Based on our survey, we introduce a detailed taxonomy of eight types of leakage, ranging from textbook errors to open research problems. We propose that researchers test for each type of leakage by filling out model info sheets, which we introduce. Finally, we conduct a reproducibility study of civil war prediction, where complex ML models are believed to vastly outperform traditional statistical models such as logistic regression (LR). When the errors are corrected, complex ML models do not perform substantively better than decades-old LR models.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science and Center for Information Technology Policy, Princeton University, Princeton, NJ 08540, USA.
Förderung
- National Science Foundation
Zitieren
APA
Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. https://doi.org/10.1016/j.patter.2023.100804
Vancouver
Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns. 2023;4(9):100804. doi:10.1016/j.patter.2023.100804
BibTeX
@article{Kapoor2023,
author = {Kapoor, Sayash and Narayanan, Arvind},
title = {Leakage and the reproducibility crisis in machine-learning-based science},
journal = {Patterns},
year = {2023},
volume = {4},
number = {9},
pages = {100804},
doi = {10.1016/j.patter.2023.100804}
}
EUVIMED