Binarized neural networks converge toward algorithmic simplicity: empirical support for the learning-as-compression hypothesis
Frontiers in Computational Neuroscience · 29.05.2026 · 20
DOI: 10.3389/fncom.2026.1791546 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Computational Neuroscience, online erschienen am 29.05.2026. Verfasst von Eduardo Y Sakabe und 6 weiteren Autor:innen. Beteiligte Einrichtung: Faculdade de Engenharia Elétrica e de Computação (FEEC), Universidade Estadual de Campinas (UNICAMP), Campinas, Brazil. Die Arbeit nennt 57 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Understanding and controlling the complexity of neural networks is a central challenge in machine learning, with implications for generalization, optimization, and model capacity. While most approaches rely on entropy-based loss functions and statistical metrics, these measures often fail to capture deeper, causally relevant algorithmic regularities embedded in network structure. We propose a shift toward algorithmic information theory, using binarized neural networks (BNNs) as a first proxy. Grounded in algorithmic probability (AP) and the universal distribution it defines, our approach characterizes learning dynamics through a formal, causally grounded lens. We apply the Block Decomposition Method (BDM), a scalable approximation of algorithmic complexity based on AP, and demonstrate that it more closely tracks structural changes during training than entropy, generally exhibiting stronger correlations with training loss across a wide range of architectures, datasets, and randomized training runs. These results support the view of training in BNNs as a process of algorithmic compression, where learning corresponds to the progressive internalization of structured regularities. In doing so, our work offers a principled estimate of learning progression and suggests a framework for complexity-aware learning and regularization, grounded in first principles from information theory, complexity, and computability.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Faculdade de Engenharia Elétrica e de Computação (FEEC), Universidade Estadual de Campinas (UNICAMP), Campinas, Brazil.
- Algorithmic Dynamics Lab, King's College London, London, United Kingdom.
- Centro de Lógica, Epistemologia e História da Ciência (CLE), Universidade Estadual de Campinas (UNICAMP), Campinas, Brazil.
- Oxford Immune Algorithmics, Oxford University Innovation & London Institute for Healthcare Engineering, London, United Kingdom.
- Data Extreme Lab (DEXL), National Laboratory for Scientific Computing (LNCC), Petrópolis, Brazil.
- São Paulo State University (UNESP), Institute of Science and Technology, Sorocaba, Brazil.
Zitieren
APA
Sakabe, E. Y., Abrahão, F. S., Simões, A., Colombini, E., Costa, P., Gudwin, R., & Zenil, H. (2026). Binarized neural networks converge toward algorithmic simplicity: empirical support for the learning-as-compression hypothesis. Frontiers in Computational Neuroscience, 20. https://doi.org/10.3389/fncom.2026.1791546
Vancouver
Sakabe EY, Abrahão FS, Simões A, Colombini E, Costa P, Gudwin R, et al. Binarized neural networks converge toward algorithmic simplicity: empirical support for the learning-as-compression hypothesis. Frontiers in Computational Neuroscience. 2026;20. doi:10.3389/fncom.2026.1791546
BibTeX
@article{Sakabe2026,
author = {Sakabe, Eduardo Y and Abrahão, Felipe S and Simões, Alexandre and Colombini, Esther and Costa, Paula and Gudwin, Ricardo and Zenil, Hector},
title = {Binarized neural networks converge toward algorithmic simplicity: empirical support for the learning-as-compression hypothesis},
journal = {Frontiers in Computational Neuroscience},
year = {2026},
volume = {20},
doi = {10.3389/fncom.2026.1791546}
}
EUVIMED