To Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review
Entropy · 12.03.2024 · 26 (3) · 252
DOI: 10.3390/e26030252 ↗Kurz gefasst
Übersichtsarbeit in der Zeitschrift Entropy, online erschienen am 12.03.2024. Verfasst von Ravid Shwartz Ziv und 1 weiteren Autor:innen. Beteiligte Einrichtung: Center of Data Science, New York University, New York, NY 10011, USA. Die Arbeit wurde bisher 78-mal zitiert (Crossref) und nennt 196 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Deep neural networks excel in supervised learning tasks but are constrained by the need for extensive labeled data. Self-supervised learning emerges as a promising alternative, allowing models to learn without explicit labels. Information theory has shaped deep neural networks, particularly the information bottleneck principle. This principle optimizes the trade-off between compression and preserving relevant information, providing a foundation for efficient network design in supervised contexts. However, its precise role and adaptation in self-supervised learning remain unclear. In this work, we scrutinize various self-supervised learning approaches from an information-theoretic perspective, introducing a unified framework that encapsulates the self-supervised information-theoretic learning problem. This framework includes multiple encoders and decoders, suggesting that all existing work on self-supervised learning can be seen as specific instances. We aim to unify these approaches to understand their underlying principles better and address the main challenge: many works present different frameworks with differing theories that may seem contradictory. By weaving existing research into a cohesive narrative, we delve into contemporary self-supervised methodologies, spotlight potential research areas, and highlight inherent challenges. Moreover, we discuss how to estimate information-theoretic quantities and their associated empirical problems. Overall, this paper provides a comprehensive review of the intersection of information theory, self-supervised learning, and deep neural networks, aiming for a better understanding through our proposed unified approach.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Center of Data Science, New York University, New York, NY 10011, USA.
- FAIR at Meta, Broadway, New York, NY 10003, USA.
Zitieren
APA
Shwartz Ziv, R., & LeCun, Y. (2024). To Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review. Entropy, 26(3), 252. https://doi.org/10.3390/e26030252
Vancouver
Shwartz Ziv R, LeCun Y. To Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review. Entropy. 2024;26(3):252. doi:10.3390/e26030252
BibTeX
@article{ShwartzZiv2024,
author = {Shwartz Ziv, Ravid and LeCun, Yann},
title = {To Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review},
journal = {Entropy},
year = {2024},
volume = {26},
number = {3},
pages = {252},
doi = {10.3390/e26030252}
}
EUVIMED