EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

How to use independent validation in Python

Thede von Oertzen, Hannes Diemerling, Timo von Oertzen

Methodology · 30.06.2026 · 22 (2) · 151-171

DOI: 10.5964/meth.18873 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Methodology, online erschienen am 30.06.2026. Verfasst von Thede von Oertzen und 2 weiteren Autor:innen. Die Arbeit nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

To statistically test whether two groups or models differ, classifier accuracy is compared. However, common accuracy estimates like cross-validation have unknown distributions, making them unsuitable for statistical inference. Alternatives like permutation tests or train-test splits are computationally expensive and limited to frequentist tests against chance. Independent Validation (IV) is a more flexible alternative providing a known estimate distribution. This enables both conventional hypothesis testing and Bayesian analysis of classifier performance. Although Python is most widely used for machine learning, a Python implementation of IV has been lacking so far. This article introduces such an implementation; beyond the core IV algorithm, the package allows to: (1) plot accuracy against training set size, (2) estimate the posterior distribution of the asymptotic accuracy, and (3) query the posterior for statistics and credible intervals. This makes it easy to apply IV when comparing accuracy posteriors across classes, datasets, or classifiers on the same data.

Zitieren

APA

Oertzen, T. V., Diemerling, H., & Oertzen, T. V. (2026). How to use independent validation in Python. Methodology, 22(2), 151-171. https://doi.org/10.5964/meth.18873

Vancouver

Oertzen TV, Diemerling H, Oertzen TV. How to use independent validation in Python. Methodology. 2026;22(2):151-171. doi:10.5964/meth.18873

BibTeX
@article{Oertzen2026,
  author = {Oertzen, Thede von and Diemerling, Hannes and Oertzen, Timo von},
  title = {How to use independent validation in Python},
  journal = {Methodology},
  year = {2026},
  volume = {22},
  number = {2},
  pages = {151--171},
  doi = {10.5964/meth.18873}
}
Kostenlos starten →