EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

MDL-AE: Investigating the trade-off between compressive fidelity and discriminative utility in self-supervised learning

Zaryab Rahman, Mattia Ottoborgo

Computing and Artificial Intelligence · 02.08.2025 · 3 (3)

DOI: 10.59400/cai3914 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computing and Artificial Intelligence, online erschienen am 02.08.2025. Verfasst von Zaryab Rahman und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Current paradigms in Self-Supervised Learning (SSL) achieve state-of-the-art results through complex, heuristic-driven pretext tasks like contrastive learning or masked image modeling. We propose a departure from these heuristics by reframing SSL through the fundamental Minimum Description Length (MDL) principle. We introduce the MDL-Autoencoder (MDL-AE), learning visual representations by optimizing a Vector Quantized Variational AutoEncoder (VQ-VAE)-based objective for efficient, discrete compression of visual data. Through rigorous experiments on the Canadian Institute for Advanced Research 10 (CIFAR-10), we demonstrate that this compression-driven objective learns a rich vocabulary of local visual concepts. However, we uncover a critical architectural insight: despite learning a visibly superior, higher-fidelity vocabulary, a more powerful tokenizer fails to improve downstream performance. We show that the MDL-AE learns holistic object parts rather than generic, composable primitives. Consequently, a sophisticated Vision Transformer (ViT) head consistently fails to outperform a simple linear probe on the flattened feature map. This architectural mismatch reveals that the nature of the learned representation dictates the optimal downstream architecture. To validate this, we demonstrate that a dedicated self-supervised alignment task, based on Masked Autoencoding of the discrete tokens, resolves this mismatch and dramatically improves performance, bridging the gap between generative fidelity and discriminative utility. Our work provides a compelling case study on co-designing objectives and downstream architectures.

Zitieren

APA

Rahman, Z., & Ottoborgo, M. (2025). MDL-AE: Investigating the trade-off between compressive fidelity and discriminative utility in self-supervised learning. Computing and Artificial Intelligence, 3(3). https://doi.org/10.59400/cai3914

Vancouver

Rahman Z, Ottoborgo M. MDL-AE: Investigating the trade-off between compressive fidelity and discriminative utility in self-supervised learning. Computing and Artificial Intelligence. 2025;3(3). doi:10.59400/cai3914

BibTeX
@article{Rahman2025,
  author = {Rahman, Zaryab and Ottoborgo, Mattia},
  title = {MDL-AE: Investigating the trade-off between compressive fidelity and discriminative utility in self-supervised learning},
  journal = {Computing and Artificial Intelligence},
  year = {2025},
  volume = {3},
  number = {3},
  doi = {10.59400/cai3914}
}
Kostenlos starten →