EUVIMEDEUVIMED entdecken
Crossref

Numerical Pruning for Efficient Autoregressive Models

Xuan Shen, Zhao Song, Yufa Zhou, Bo Chen, Jing Liu, Ruiyi Zhang, Ryan A. Rossi, Hao Tan, Tong Yu, Xiang Chen, Yufan Zhou, Tong Sun und 3 weitere

Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (19) · 20418-20426

DOI: 10.1609/aaai.v39i19.34249 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Xuan Shen und 14 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Transformers have emerged as the leading architecture in deep learning, proving to be versatile and highly effective across diverse domains beyond language and image processing. However, their impressive performance often incurs high computational costs due to their substantial model size. This paper focuses on compressing decoder-only transformer-based autoregressive models through structural weight pruning to improve the model efficiency while preserving performance for both language and image generation tasks. Specifically, we propose a training-free pruning method that calculates a numerical score with Newton's method for the Attention and MLP modules, respectively. Besides, we further propose another compensation algorithm to recover the pruned model for better performance. To verify the effectiveness of our method, we provide both theoretical support and extensive experiments. Our experiments show that our method achieves state-of-the-art performance with reduced memory usage and faster generation speeds on GPUs.

Zitieren

APA

Shen, X., Song, Z., Zhou, Y., Chen, B., Liu, J., Zhang, R., Rossi, R. A., Tan, H., Yu, T., Chen, X., Zhou, Y., Sun, T., Zhao, P., Wang, Y., & Gu, J. (2025). Numerical Pruning for Efficient Autoregressive Models. Proceedings of the AAAI Conference on Artificial Intelligence, 39(19), 20418-20426. https://doi.org/10.1609/aaai.v39i19.34249

Vancouver

Shen X, Song Z, Zhou Y, Chen B, Liu J, Zhang R, et al. Numerical Pruning for Efficient Autoregressive Models. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(19):20418-20426. doi:10.1609/aaai.v39i19.34249

BibTeX
@article{Shen2025,
  author = {Shen, Xuan and Song, Zhao and Zhou, Yufa and Chen, Bo and Liu, Jing and Zhang, Ruiyi and Rossi, Ryan A. and Tan, Hao and Yu, Tong and Chen, Xiang and Zhou, Yufan and Sun, Tong and Zhao, Pu and Wang, Yanzhi and Gu, Jiuxiang},
  title = {Numerical Pruning for Efficient Autoregressive Models},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2025},
  volume = {39},
  number = {19},
  pages = {20418--20426},
  doi = {10.1609/aaai.v39i19.34249}
}
Kostenlos starten →