What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph
Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (4) · 4075-4083
DOI: 10.1609/aaai.v39i4.32427 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Yutao Jiang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref).
Abstract (Originalsprache)
Recent Multimodal Large Language Models(MLLMs) often use a large number of visual tokens to compensate their visual shortcoming, leading to excessive computation and obvious visual redundancy. In this paper, we investigate what kind of visual tokens are needed for MLLMs, and reveal that both foreground and background tokens are critical for MLLMs given the varying difficulties of examples. Based on this observation, we propose a graph-based method towards training-free visual token pruning, termed G-Prune. In particular, G-Prune regards visual tokens as nodes, and construct their connections based on their semantic similarities. Afterwards, the information flow is propagated via weighted links, and the most important tokens after iterations are kept for MLLMs, which can be front or background. To validate G-Prune, we apply it to a recent MLLM called LLaVA-NeXT, and conduct extensive experiments on a set of benchmarks. The experiment results show that G-Prune can greatly reduce computation overhead while retaining high performance on both coarse- and fine-grained tasks. For instance, G-Prune can reduce 63.57% FLOPs of LLaVA-NeXT on VQA2.0 and TextVQA with only 0.95% and 2.34% accuracy drops, respectively.
Zitieren
APA
Jiang, Y., Wu, Q., Lin, W., Yu, W., & Zhou, Y. (2025). What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph. Proceedings of the AAAI Conference on Artificial Intelligence, 39(4), 4075-4083. https://doi.org/10.1609/aaai.v39i4.32427
Vancouver
Jiang Y, Wu Q, Lin W, Yu W, Zhou Y. What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(4):4075-4083. doi:10.1609/aaai.v39i4.32427
BibTeX
@article{Jiang2025,
author = {Jiang, Yutao and Wu, Qiong and Lin, Wenhao and Yu, Wei and Zhou, Yiyi},
title = {What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2025},
volume = {39},
number = {4},
pages = {4075--4083},
doi = {10.1609/aaai.v39i4.32427}
}
EUVIMED