Not All Attention is Needed: Parameter and Computation Efficient Tuning for Multi-modal Large Language Models via Effective Attention Skipping
International Journal of Computer Vision · 2026-03 · 134 (3)
DOI: 10.1007/s11263-025-02702-1 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 21.02.2026. Verfasst von Qiong Wu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 81 Literaturangaben. Gefördert von: National Science Fund for Distinguished Young Scholars, National Natural Science Foundation of China, Natural Science Foundation of Fujian Province of China.
Abstract (Originalsprache)
Recently, Multi-modal Large Language Models (MLLMs) have garnered an influx of interest from both academia and industry. Despite great progress, MLLMs still have a high demand of downstream task tuning for applications, which consumes excessive parameter and computation overhead. In this paper, we propose a novel parameter and computation efficient tuning method for MLLMs, termed Effective Attention Skipping (EAS). Concretely, we first reveal that multi-head attentions (MHAs) in MLLMs, the primary source of computation, are often redundant to downstream tasks. Based on this observation, EAS evaluates attention redundancy and skips the less important MHAs to speed up inference. Besides, we also propose a novel propagation-of-information adapter (PIA) to serve attention skipping while maintaining parameter efficiency. More importantly, PIA can be further re-parameterized into feed-forward networks (FFNs) for zero-extra latency. To validate EAS, we apply it to three common MLLMs, and conduct extensive experiments on six downstream tasks. The experimental results show that EAS can not only retain the high performance of MLLMs but also reduce the updated parameters scale greatly, while speeding up inference speed to a large extent. For instance, LLaVA-EAS can obtain 96.2% accuracy while accelerating the inference speed by about +20% on ScienceQA. Our code is publicly released at https://github.com/DoubtedSteam/EAS.
Förderung
- National Science Fund for Distinguished Young Scholars
- National Natural Science Foundation of China
- Natural Science Foundation of Fujian Province of China
- Fundamental Research Funds for the Central Universities
Zitieren
APA
Wu, Q., Zhou, Y., Ye, W., Sun, X., & Ji, R. (2026). Not All Attention is Needed: Parameter and Computation Efficient Tuning for Multi-modal Large Language Models via Effective Attention Skipping. International Journal of Computer Vision, 134(3). https://doi.org/10.1007/s11263-025-02702-1
Vancouver
Wu Q, Zhou Y, Ye W, Sun X, Ji R. Not All Attention is Needed: Parameter and Computation Efficient Tuning for Multi-modal Large Language Models via Effective Attention Skipping. International Journal of Computer Vision. 2026;134(3). doi:10.1007/s11263-025-02702-1
BibTeX
@article{Wu2026,
author = {Wu, Qiong and Zhou, Yiyi and Ye, Weihao and Sun, Xiaoshuai and Ji, Rongrong},
title = {Not All Attention is Needed: Parameter and Computation Efficient Tuning for Multi-modal Large Language Models via Effective Attention Skipping},
journal = {International Journal of Computer Vision},
year = {2026},
volume = {134},
number = {3},
doi = {10.1007/s11263-025-02702-1}
}
EUVIMED