MDL-AE: Investigating the trade-off between compressive fidelity and discriminative utility in self-supervised learning
Computing and Artificial Intelligence · 02.08.2025 · 3 (3)
DOI: 10.59400/cai3914 ↗Kurz gefasst
Fachartikel in der Zeitschrift Computing and Artificial Intelligence, online erschienen am 02.08.2025. Verfasst von Zaryab Rahman und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Current paradigms in Self-Supervised Learning (SSL) achieve state-of-the-art results through complex, heuristic-driven pretext tasks like contrastive learning or masked image modeling. We propose a departure from these heuristics by reframing SSL through the fundamental Minimum Description Length (MDL) principle. We introduce the MDL-Autoencoder (MDL-AE), learning visual representations by optimizing a Vector Quantized Variational AutoEncoder (VQ-VAE)-based objective for efficient, discrete compression of visual data. Through rigorous experiments on the Canadian Institute for Advanced Research 10 (CIFAR-10), we demonstrate that this compression-driven objective learns a rich vocabulary of local visual concepts. However, we uncover a critical architectural insight: despite learning a visibly superior, higher-fidelity vocabulary, a more powerful tokenizer fails to improve downstream performance. We show that the MDL-AE learns holistic object parts rather than generic, composable primitives. Consequently, a sophisticated Vision Transformer (ViT) head consistently fails to outperform a simple linear probe on the flattened feature map. This architectural mismatch reveals that the nature of the learned representation dictates the optimal downstream architecture. To validate this, we demonstrate that a dedicated self-supervised alignment task, based on Masked Autoencoding of the discrete tokens, resolves this mismatch and dramatically improves performance, bridging the gap between generative fidelity and discriminative utility. Our work provides a compelling case study on co-designing objectives and downstream architectures.
Zitieren
APA
Rahman, Z., & Ottoborgo, M. (2025). MDL-AE: Investigating the trade-off between compressive fidelity and discriminative utility in self-supervised learning. Computing and Artificial Intelligence, 3(3). https://doi.org/10.59400/cai3914
Vancouver
Rahman Z, Ottoborgo M. MDL-AE: Investigating the trade-off between compressive fidelity and discriminative utility in self-supervised learning. Computing and Artificial Intelligence. 2025;3(3). doi:10.59400/cai3914
BibTeX
@article{Rahman2025,
author = {Rahman, Zaryab and Ottoborgo, Mattia},
title = {MDL-AE: Investigating the trade-off between compressive fidelity and discriminative utility in self-supervised learning},
journal = {Computing and Artificial Intelligence},
year = {2025},
volume = {3},
number = {3},
doi = {10.59400/cai3914}
}
EUVIMED