EUVIMEDEUVIMED entdecken
Crossref

Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference

Zhihang Lin, Mingbao Lin, Luxi Lin, Rongrong Ji

Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (5) · 5334-5342

DOI: 10.1609/aaai.v39i5.32567 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Zhihang Lin und 3 weiteren Autor:innen. Die Arbeit wurde bisher 54-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal large language models (MLLMs) demand considerable computations for inference due to the extensive parameters and the additional input tokens needed for visual information representation. Herein, we introduce Visual Tokens Withdrawal (VTW), a plug-and-play module to boost MLLMs for rapid inference. Our approach is inspired by two intriguing phenomena we have observed: (1) the attention sink phenomenon that is prevalent in LLMs also persists in MLLMs, suggesting that initial tokens and nearest tokens receive the majority of attention, while middle vision tokens garner minimal attention in deep layers; (2) the presence of information migration, which implies that visual information is transferred to subsequent text tokens within the first few layers of MLLMs. As per our findings, we conclude that vision tokens are unnecessary in the deep layers of MLLMs. Thus, we strategically withdraw them at a certain layer, enabling only text tokens to engage in subsequent layers. To pinpoint the ideal layer for VTW, we initially analyze a limited set of tiny datasets and choose the first layer that meets the Kullback-Leibler divergence criterion. Our VTW approach can cut computational overhead by over 40% across diverse multimodal tasks while maintaining performance.

Zitieren

APA

Lin, Z., Lin, M., Lin, L., & Ji, R. (2025). Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference. Proceedings of the AAAI Conference on Artificial Intelligence, 39(5), 5334-5342. https://doi.org/10.1609/aaai.v39i5.32567

Vancouver

Lin Z, Lin M, Lin L, Ji R. Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(5):5334-5342. doi:10.1609/aaai.v39i5.32567

BibTeX
@article{Lin2025,
  author = {Lin, Zhihang and Lin, Mingbao and Lin, Luxi and Ji, Rongrong},
  title = {Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2025},
  volume = {39},
  number = {5},
  pages = {5334--5342},
  doi = {10.1609/aaai.v39i5.32567}
}
Kostenlos starten →