EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Multi-dimensional collaborative pruning: optimizing vision transformers via token, attention head and FFN neuron

Yalong Liu, Tiebing Wang, Xinchang Zhang, Lin Wang, Yu Zhao, Changtong Ding, Quanrun Song

Journal of King Saud University Computer and Information Sciences · 2026-07 · 38 (5)

DOI: 10.1007/s44443-026-00607-8 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of King Saud University Computer and Information Sciences, online erschienen am 02.06.2026. Verfasst von Yalong Liu und 6 weiteren Autor:innen. Die Arbeit nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Natural Science Foundation of China, Shandong Province Key R&D Program (Innovation Capability Enhancement Project for Technology based Small and Medium sized Enterprises) Project.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recently, Vision Transformers (ViTs) have gained increasing attention. Research shows the ViT models perform well on large datasets. Yet, high computation limits their deployment on low-resource devices. Investigations have revealed that, due to the sparse nature of attention, only a subset of tokens contribute to the final prediction. Consequently, some existing ViT compression methods aim to alleviate the model burden by eliminating unimportant tokens. Nevertheless, optimizing the model solely through token sparsification is insufficient. And the excessively high token sparsity ratio may lead to considerable accuracy degradation. Moreover, the computation of the attention head module and the feed-forward network (FFN) module in each layer of the model still accounts for a considerable proportion of the overall computational load. In response to the aforementioned challenges, this paper presents a novel pruning method that operates across three dimensions: token, attention head, and FFN neuron. Initially, an attention-gradient importance analysis method is introduced to quantify the importance scores of tokens. Subsequently, fusion pruning is executed based on these token scores. Concurrently, a multi-dimensional optimization searcher is proposed. It identifies the optimal pruning strategies for attention head and FFN neuron across different layers. This enables the pruning rate of each layer is different, and it facilitates hierarchical pruning. By integrating the pruning efforts across these three dimensions, this paper achieves efficient model compression while minimizing the degradation of model performance. Experimental results demonstrate that our method significantly reduces the computational cost of the ViT model. Specifically, for Deit-small, it achieves a nearly 50%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\%$$\end{document} reduction in FLOPs with only 1.2%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\%$$\end{document} accuracy loss. For Deit-base, it achieves a nearly 44%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\%$$\end{document} reduction in FLOPs with just a 0.9%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\%$$\end{document} accuracy loss.

Förderung

  • National Natural Science Foundation of China
  • Shandong Province Key R&D Program (Innovation Capability Enhancement Project for Technology based Small and Medium sized Enterprises) Project

Zitieren

APA

Liu, Y., Wang, T., Zhang, X., Wang, L., Zhao, Y., Ding, C., & Song, Q. (2026). Multi-dimensional collaborative pruning: optimizing vision transformers via token, attention head and FFN neuron. Journal of King Saud University Computer and Information Sciences, 38(5). https://doi.org/10.1007/s44443-026-00607-8

Vancouver

Liu Y, Wang T, Zhang X, Wang L, Zhao Y, Ding C, et al. Multi-dimensional collaborative pruning: optimizing vision transformers via token, attention head and FFN neuron. Journal of King Saud University Computer and Information Sciences. 2026;38(5). doi:10.1007/s44443-026-00607-8

BibTeX
@article{Liu2026,
  author = {Liu, Yalong and Wang, Tiebing and Zhang, Xinchang and Wang, Lin and Zhao, Yu and Ding, Changtong and Song, Quanrun},
  title = {Multi-dimensional collaborative pruning: optimizing vision transformers via token, attention head and FFN neuron},
  journal = {Journal of King Saud University Computer and Information Sciences},
  year = {2026},
  volume = {38},
  number = {5},
  doi = {10.1007/s44443-026-00607-8}
}
Kostenlos starten →