EUVIMEDObjevte EUVIMED
Open access · CC BY 4.0

How to use independent validation in Python

Thede von Oertzen, Hannes Diemerling, Timo von Oertzen

Methodology · 2026-06-30 · 22 (2) · 151-171

DOI: 10.5964/meth.18873 ↗

At a glance

Journal article in Methodology, published online on 2026-06-30. Written by Thede von Oertzen and 2 co-authors. 38 references. The full text is freely available (open access, CC BY 4.0).

AbstraktBibliografické údajeCiteRelated studies

Abstract (original language)

To statistically test whether two groups or models differ, classifier accuracy is compared. However, common accuracy estimates like cross-validation have unknown distributions, making them unsuitable for statistical inference. Alternatives like permutation tests or train-test splits are computationally expensive and limited to frequentist tests against chance. Independent Validation (IV) is a more flexible alternative providing a known estimate distribution. This enables both conventional hypothesis testing and Bayesian analysis of classifier performance. Although Python is most widely used for machine learning, a Python implementation of IV has been lacking so far. This article introduces such an implementation; beyond the core IV algorithm, the package allows to: (1) plot accuracy against training set size, (2) estimate the posterior distribution of the asymptotic accuracy, and (3) query the posterior for statistics and credible intervals. This makes it easy to apply IV when comparing accuracy posteriors across classes, datasets, or classifiers on the same data.

Cite

APA

Oertzen, T. V., Diemerling, H., & Oertzen, T. V. (2026). How to use independent validation in Python. Methodology, 22(2), 151-171. https://doi.org/10.5964/meth.18873

Vancouver

Oertzen TV, Diemerling H, Oertzen TV. How to use independent validation in Python. Methodology. 2026;22(2):151-171. doi:10.5964/meth.18873

BibTeX
@article{Oertzen2026,
  author = {Oertzen, Thede von and Diemerling, Hannes and Oertzen, Timo von},
  title = {How to use independent validation in Python},
  journal = {Methodology},
  year = {2026},
  volume = {22},
  number = {2},
  pages = {151--171},
  doi = {10.5964/meth.18873}
}
Start for free →