EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Binarized neural networks converge toward algorithmic simplicity: empirical support for the learning-as-compression hypothesis

Eduardo Y Sakabe, Felipe S Abrahão, Alexandre Simões, Esther Colombini, Paula Costa, Ricardo Gudwin, Hector Zenil

Frontiers in Computational Neuroscience · 29.05.2026 · 20

DOI: 10.3389/fncom.2026.1791546 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Computational Neuroscience, online erschienen am 29.05.2026. Verfasst von Eduardo Y Sakabe und 6 weiteren Autor:innen. Beteiligte Einrichtung: Faculdade de Engenharia Elétrica e de Computação (FEEC), Universidade Estadual de Campinas (UNICAMP), Campinas, Brazil. Die Arbeit nennt 57 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Understanding and controlling the complexity of neural networks is a central challenge in machine learning, with implications for generalization, optimization, and model capacity. While most approaches rely on entropy-based loss functions and statistical metrics, these measures often fail to capture deeper, causally relevant algorithmic regularities embedded in network structure. We propose a shift toward algorithmic information theory, using binarized neural networks (BNNs) as a first proxy. Grounded in algorithmic probability (AP) and the universal distribution it defines, our approach characterizes learning dynamics through a formal, causally grounded lens. We apply the Block Decomposition Method (BDM), a scalable approximation of algorithmic complexity based on AP, and demonstrate that it more closely tracks structural changes during training than entropy, generally exhibiting stronger correlations with training loss across a wide range of architectures, datasets, and randomized training runs. These results support the view of training in BNNs as a process of algorithmic compression, where learning corresponds to the progressive internalization of structured regularities. In doing so, our work offers a principled estimate of learning progression and suggests a framework for complexity-aware learning and regularization, grounded in first principles from information theory, complexity, and computability.

Schlagwörter

Stichwörter der Autor:innen

Block Decomposition MethodShannon entropyalgorithmic information theorybinarized neural networkslearning as compression

Autor:innen und Institutionen

  • Faculdade de Engenharia Elétrica e de Computação (FEEC), Universidade Estadual de Campinas (UNICAMP), Campinas, Brazil.
  • Algorithmic Dynamics Lab, King's College London, London, United Kingdom.
  • Centro de Lógica, Epistemologia e História da Ciência (CLE), Universidade Estadual de Campinas (UNICAMP), Campinas, Brazil.
  • Oxford Immune Algorithmics, Oxford University Innovation & London Institute for Healthcare Engineering, London, United Kingdom.
  • Data Extreme Lab (DEXL), National Laboratory for Scientific Computing (LNCC), Petrópolis, Brazil.
  • São Paulo State University (UNESP), Institute of Science and Technology, Sorocaba, Brazil.

Zitieren

APA

Sakabe, E. Y., Abrahão, F. S., Simões, A., Colombini, E., Costa, P., Gudwin, R., & Zenil, H. (2026). Binarized neural networks converge toward algorithmic simplicity: empirical support for the learning-as-compression hypothesis. Frontiers in Computational Neuroscience, 20. https://doi.org/10.3389/fncom.2026.1791546

Vancouver

Sakabe EY, Abrahão FS, Simões A, Colombini E, Costa P, Gudwin R, et al. Binarized neural networks converge toward algorithmic simplicity: empirical support for the learning-as-compression hypothesis. Frontiers in Computational Neuroscience. 2026;20. doi:10.3389/fncom.2026.1791546

BibTeX
@article{Sakabe2026,
  author = {Sakabe, Eduardo Y and Abrahão, Felipe S and Simões, Alexandre and Colombini, Esther and Costa, Paula and Gudwin, Ricardo and Zenil, Hector},
  title = {Binarized neural networks converge toward algorithmic simplicity: empirical support for the learning-as-compression hypothesis},
  journal = {Frontiers in Computational Neuroscience},
  year = {2026},
  volume = {20},
  doi = {10.3389/fncom.2026.1791546}
}
Kostenlos starten →