EUVIMEDEUVIMED entdecken
Crossref

Compressing Transformers: Features Are Low-Rank, but Weights Are Not!

Hao Yu, Jianxin Wu

Proceedings of the AAAI Conference on Artificial Intelligence · 26.06.2023 · 37 (9) · 11007-11015

DOI: 10.1609/aaai.v37i9.26304 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 26.06.2023. Verfasst von Hao Yu und 1 weiteren Autor:innen. Die Arbeit wurde bisher 23-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Transformer and its variants achieve excellent results in various computer vision and natural language processing tasks, but high computational costs and reliance on large training datasets restrict their deployment in resource-constrained settings. Low-rank approximation of model weights has been effective in compressing CNN models, but its application to transformers has been less explored and is less effective. Existing methods require the complete dataset to fine-tune compressed models, which are both time-consuming and data-hungry. This paper reveals that the features (i.e., activations) are low-rank, but model weights are surprisingly not low-rank. Hence, AAFM is proposed, which adaptively determines the compressed model structure and locally compresses each linear layer's output features rather than the model weights. A second stage, GFM, optimizes the entire compressed network holistically. Both AAFM and GFM only use few training samples without labels, that is, they are few-shot, unsupervised, fast and effective. For example, with only 2K images without labels, 33% of the parameters are removed in DeiT-B with 18.8% relative throughput increase, but only a 0.23% accuracy loss for ImageNet recognition. The proposed methods are successfully applied to the language modeling task in NLP, too. Besides, the few-shot compressed models generalize well in downstream tasks.

Zitieren

APA

Yu, H., & Wu, J. (2023). Compressing Transformers: Features Are Low-Rank, but Weights Are Not!. Proceedings of the AAAI Conference on Artificial Intelligence, 37(9), 11007-11015. https://doi.org/10.1609/aaai.v37i9.26304

Vancouver

Yu H, Wu J. Compressing Transformers: Features Are Low-Rank, but Weights Are Not!. Proceedings of the AAAI Conference on Artificial Intelligence. 2023;37(9):11007-11015. doi:10.1609/aaai.v37i9.26304

BibTeX
@article{Yu2023,
  author = {Yu, Hao and Wu, Jianxin},
  title = {Compressing Transformers: Features Are Low-Rank, but Weights Are Not!},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2023},
  volume = {37},
  number = {9},
  pages = {11007--11015},
  doi = {10.1609/aaai.v37i9.26304}
}
Kostenlos starten →