EUVIMEDEUVIMED entdecken
ÜbersichtsarbeitOpen Access · CC BY 4.0

To Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review

Ravid Shwartz Ziv, Yann LeCun

Entropy · 12.03.2024 · 26 (3) · 252

DOI: 10.3390/e26030252 ↗

Kurz gefasst

Übersichtsarbeit in der Zeitschrift Entropy, online erschienen am 12.03.2024. Verfasst von Ravid Shwartz Ziv und 1 weiteren Autor:innen. Beteiligte Einrichtung: Center of Data Science, New York University, New York, NY 10011, USA. Die Arbeit wurde bisher 78-mal zitiert (Crossref) und nennt 196 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Deep neural networks excel in supervised learning tasks but are constrained by the need for extensive labeled data. Self-supervised learning emerges as a promising alternative, allowing models to learn without explicit labels. Information theory has shaped deep neural networks, particularly the information bottleneck principle. This principle optimizes the trade-off between compression and preserving relevant information, providing a foundation for efficient network design in supervised contexts. However, its precise role and adaptation in self-supervised learning remain unclear. In this work, we scrutinize various self-supervised learning approaches from an information-theoretic perspective, introducing a unified framework that encapsulates the self-supervised information-theoretic learning problem. This framework includes multiple encoders and decoders, suggesting that all existing work on self-supervised learning can be seen as specific instances. We aim to unify these approaches to understand their underlying principles better and address the main challenge: many works present different frameworks with differing theories that may seem contradictory. By weaving existing research into a cohesive narrative, we delve into contemporary self-supervised methodologies, spotlight potential research areas, and highlight inherent challenges. Moreover, we discuss how to estimate information-theoretic quantities and their associated empirical problems. Overall, this paper provides a comprehensive review of the intersection of information theory, self-supervised learning, and deep neural networks, aiming for a better understanding through our proposed unified approach.

Schlagwörter

Stichwörter der Autor:innen

deep neural networksinformation theoryrepresentation learningself-supervised learning

Autor:innen und Institutionen

  • Center of Data Science, New York University, New York, NY 10011, USA.
  • FAIR at Meta, Broadway, New York, NY 10003, USA.

Zitieren

APA

Shwartz Ziv, R., & LeCun, Y. (2024). To Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review. Entropy, 26(3), 252. https://doi.org/10.3390/e26030252

Vancouver

Shwartz Ziv R, LeCun Y. To Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review. Entropy. 2024;26(3):252. doi:10.3390/e26030252

BibTeX
@article{ShwartzZiv2024,
  author = {Shwartz Ziv, Ravid and LeCun, Yann},
  title = {To Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review},
  journal = {Entropy},
  year = {2024},
  volume = {26},
  number = {3},
  pages = {252},
  doi = {10.3390/e26030252}
}
Kostenlos starten →