EUVIMEDEUVIMED entdecken
Crossref

Tight PAC-Bayesian Risk Certificates for Contrastive Learning

Anna van Elst, Debarghya Ghoshdastidar

SIAM Journal on Mathematics of Data Science · 31.12.2025 · 7 (4) · 1904-1927

DOI: 10.1137/24m1715283 ↗

Kurz gefasst

Fachartikel in der Zeitschrift SIAM Journal on Mathematics of Data Science, online erschienen am 05.12.2025. Verfasst von Anna van Elst und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 58 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Contrastive representation learning is a modern paradigm for learning representations of unlabeled data via augmentations—precisely, contrastive models learn to embed semantically similar pairs of samples (positive pairs) closer than independently drawn samples (negative samples). In spite of its empirical success and widespread use in foundation models, statistical theory for contrastive learning remains less explored. Recent works have developed generalization error bounds for contrastive losses, but the resulting risk certificates are either vacuous (certificates based on Rademacher complexity or [Formula: see text]-divergence) or require strong assumptions about samples that are unreasonable in practice. The present paper develops nonvacuous probably approximately correct (PAC)-Bayesian risk certificates for contrastive representation learning, considering the practical considerations of the popular SimCLR framework. Notably, we take into account that SimCLR reuses positive pairs of augmented data as negative samples for other data, thereby inducing strong dependence and making classical PAC or PAC-Bayesian bounds inapplicable. We further refine existing bounds on the downstream classification loss by incorporating SimCLR-specific factors, including data augmentation and temperature scaling, and derive risk certificates for the contrastive zero-one risk. The resulting bounds for contrastive loss and downstream prediction are much tighter than those of previous risk certificates, as demonstrated by experiments on MNIST and CIFAR-10.

Zitieren

APA

Elst, A. V., & Ghoshdastidar, D. (2025). Tight PAC-Bayesian Risk Certificates for Contrastive Learning. SIAM Journal on Mathematics of Data Science, 7(4), 1904-1927. https://doi.org/10.1137/24m1715283

Vancouver

Elst AV, Ghoshdastidar D. Tight PAC-Bayesian Risk Certificates for Contrastive Learning. SIAM Journal on Mathematics of Data Science. 2025;7(4):1904-1927. doi:10.1137/24m1715283

BibTeX
@article{Elst2025,
  author = {Elst, Anna van and Ghoshdastidar, Debarghya},
  title = {Tight PAC-Bayesian Risk Certificates for Contrastive Learning},
  journal = {SIAM Journal on Mathematics of Data Science},
  year = {2025},
  volume = {7},
  number = {4},
  pages = {1904--1927},
  doi = {10.1137/24m1715283}
}
Kostenlos starten →