EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Leakage and the reproducibility crisis in machine-learning-based science

Sayash Kapoor, Arvind Narayanan

Patterns · 2023-09 · 4 (9) · 100804

DOI: 10.1016/j.patter.2023.100804 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Patterns, 2023-09. Verfasst von Sayash Kapoor und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Center for Information Technology Policy, Princeton University, Princeton, NJ 08540, USA. Die Arbeit wurde bisher 1560-mal zitiert (Crossref) und nennt 104 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Machine-learning (ML) methods have gained prominence in the quantitative sciences. However, there are many known methodological pitfalls, including data leakage, in ML-based science. We systematically investigate reproducibility issues in ML-based science. Through a survey of literature in fields that have adopted ML methods, we find 17 fields where leakage has been found, collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions. Based on our survey, we introduce a detailed taxonomy of eight types of leakage, ranging from textbook errors to open research problems. We propose that researchers test for each type of leakage by filling out model info sheets, which we introduce. Finally, we conduct a reproducibility study of civil war prediction, where complex ML models are believed to vastly outperform traditional statistical models such as logistic regression (LR). When the errors are corrected, complex ML models do not perform substantively better than decades-old LR models.

Schlagwörter

Stichwörter der Autor:innen

leakagemachine learningreproducibility

Autor:innen und Institutionen

  • Department of Computer Science and Center for Information Technology Policy, Princeton University, Princeton, NJ 08540, USA.

Förderung

  • National Science Foundation

Zitieren

APA

Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. https://doi.org/10.1016/j.patter.2023.100804

Vancouver

Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns. 2023;4(9):100804. doi:10.1016/j.patter.2023.100804

BibTeX
@article{Kapoor2023,
  author = {Kapoor, Sayash and Narayanan, Arvind},
  title = {Leakage and the reproducibility crisis in machine-learning-based science},
  journal = {Patterns},
  year = {2023},
  volume = {4},
  number = {9},
  pages = {100804},
  doi = {10.1016/j.patter.2023.100804}
}
Kostenlos starten →