Algorithmic Compression via Pretrained Neural Networks
Entropy · 27.05.2026 · 28 (6) · 596
DOI: 10.3390/e28060596 ↗Kurz gefasst
Übersichtsarbeit in der Zeitschrift Entropy, online erschienen am 27.05.2026. Verfasst von Tim Genewein und 4 weiteren Autor:innen. Beteiligte Einrichtung: Google DeepMind, London N1C 4DJ, UK. Die Arbeit nennt 63 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
The success of large neural networks trained for sequential prediction via log-loss minimization over massive and diverse datasets has sparked debate regarding the fundamental limits of this paradigm. While these models are not explicitly programmed to perform planning and search, their behavior increasingly resembles complex reasoning and adaptive problem-solving. This paper reviews a series of theoretical and empirical works, aiming to bridge the gap between the practical success of LLMs and formal theories of computation and intelligence—that is, algorithmic information theory and Universal Artificial Intelligence. Grounded in the framework of memory-based meta-learning, the main argument is that training sequence models to predict the next token across diverse tasks implicitly meta-trains them to perform algorithmic compression, thereby performing (amortized) Bayesian inference over the task in-context. Consequently, when pretrained on a sufficiently rich data distribution, the resulting neural networks behave as if compressing by inferring the generative algorithm producing the observed data. We discuss recent theoretical and empirical evidence demonstrating that this approach can approximate Solomonoff induction in the theoretical limit, match exact Bayesian inference on complex sources in practice, achieve strong compression on out-of-distribution data, and synthesize complex in-context algorithms like chessboard evaluations. As models become more capable and general, the theoretical understanding through the lens of algorithmic information theory, including hard theoretical limits and how far practical models are from them, becomes increasingly relevant. We thus conclude our paper by outlining a number of open research questions to further bridge the gap from well-understood theory to modern machine learning practice.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Google DeepMind, London N1C 4DJ, UK.
- ANU School of Computing, Australian National University, Canberra, ACT 2601, Australia.
ORCID-Profile: Tim Genewein, Jordi Grau-Moya, Li Kevin Wenliang, Laurent Orseau, Marcus Hutter
Zitieren
APA
Genewein, T., Grau-Moya, J., Wenliang, L. K., Orseau, L., & Hutter, M. (2026). Algorithmic Compression via Pretrained Neural Networks. Entropy, 28(6), 596. https://doi.org/10.3390/e28060596
Vancouver
Genewein T, Grau-Moya J, Wenliang LK, Orseau L, Hutter M. Algorithmic Compression via Pretrained Neural Networks. Entropy. 2026;28(6):596. doi:10.3390/e28060596
BibTeX
@article{Genewein2026,
author = {Genewein, Tim and Grau-Moya, Jordi and Wenliang, Li Kevin and Orseau, Laurent and Hutter, Marcus},
title = {Algorithmic Compression via Pretrained Neural Networks},
journal = {Entropy},
year = {2026},
volume = {28},
number = {6},
pages = {596},
doi = {10.3390/e28060596}
}
EUVIMED