EUVIMEDEUVIMED entdecken
Crossref

Learnable Permutation for Structured Sparsity on Transformer Models

Zekai Li, Ji Liu, Guanchen Li, Yixing Xu, Ziqiong Liu, Xuanwu Yin, Dong Li, Emad Barsoum

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (28) · 23319-23327

DOI: 10.1609/aaai.v40i28.39501 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Zekai Li und 7 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Structured sparsity has emerged as a popular model pruning technique, widely adopted in various architectures, including CNNs, Transformer models, and especially large language models (LLMs) in recent years. A promising direction to further improve post-pruning performance is weight permutation, which reorders model weights into patterns more amenable to pruning. However, the exponential growth of the permutation search space with the scale of Transformer architectures forces most methods to rely on greedy or heuristic algorithms, limiting the effectiveness of reordering. In this work, we propose a novel end-to-end learnable permutation framework. Our method introduces a learnable permutation cost matrix to quantify the cost of swapping any two input channels of a given weight matrix, a differentiable bipartite matching solver to obtain the optimal binary permutation matrix given a cost matrix, and a sparsity optimization loss function to directly optimize the permutation operator. We extensively validate our approach on vision and language Transformers, demonstrating that our method achieves state-of-the-art permutation results for structured sparsity.

Zitieren

APA

Li, Z., Liu, J., Li, G., Xu, Y., Liu, Z., Yin, X., Li, D., & Barsoum, E. (2026). Learnable Permutation for Structured Sparsity on Transformer Models. Proceedings of the AAAI Conference on Artificial Intelligence, 40(28), 23319-23327. https://doi.org/10.1609/aaai.v40i28.39501

Vancouver

Li Z, Liu J, Li G, Xu Y, Liu Z, Yin X, et al. Learnable Permutation for Structured Sparsity on Transformer Models. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(28):23319-23327. doi:10.1609/aaai.v40i28.39501

BibTeX
@article{Li2026,
  author = {Li, Zekai and Liu, Ji and Li, Guanchen and Xu, Yixing and Liu, Ziqiong and Yin, Xuanwu and Li, Dong and Barsoum, Emad},
  title = {Learnable Permutation for Structured Sparsity on Transformer Models},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {28},
  pages = {23319--23327},
  doi = {10.1609/aaai.v40i28.39501}
}
Kostenlos starten →