The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (29) · 24639-24647
DOI: 10.1609/aaai.v40i29.39648 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Hikari Otsuka und 5 weiteren Autor:innen.
Abstract (Originalsprache)
The strong lottery ticket hypothesis (SLTH) conjectures that high-performing subnetworks, called strong lottery tickets (SLTs), are hidden in randomly initialized neural networks. Although recent theoretical studies have established the SLTH across various neural architectures, the SLTH for transformer architectures still lacks theoretical understanding. In particular, the current theory of the SLTH does not yet account for the multi-head attention (MHA) mechanism, a core component of transformers. To address this gap, we introduce a theoretical analysis of the existence of SLTs within MHAs. We prove that, if a randomly initialized MHA of H heads and input dimension d has the hidden dimension O(d log(Hd^(3/2))) for the key and value, it contains an SLT that approximates an arbitrary MHA with the same input dimension with high probability. Furthermore, by leveraging this theory for MHAs, we extend the SLTH to transformers without normalization layers. We empirically validate our theoretical findings, demonstrating that the approximation error between the SLT within a source model (MHA and transformer) and an approximate target counterpart decreases exponentially by increasing the hidden dimension of the source model.
Zitieren
APA
Otsuka, H., Chijiwa, D., Okoshi, Y., Fujiki, D., Takeuchi, S., & Motomura, M. (2026). The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms. Proceedings of the AAAI Conference on Artificial Intelligence, 40(29), 24639-24647. https://doi.org/10.1609/aaai.v40i29.39648
Vancouver
Otsuka H, Chijiwa D, Okoshi Y, Fujiki D, Takeuchi S, Motomura M. The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(29):24639-24647. doi:10.1609/aaai.v40i29.39648
BibTeX
@article{Otsuka2026,
author = {Otsuka, Hikari and Chijiwa, Daiki and Okoshi, Yasuyuki and Fujiki, Daichi and Takeuchi, Susumu and Motomura, Masato},
title = {The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {29},
pages = {24639--24647},
doi = {10.1609/aaai.v40i29.39648}
}
EUVIMED