HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models
Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (2) · 1773-1781
DOI: 10.1609/aaai.v39i2.32171 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Kazi Hasan Ibn Arif und 5 weiteren Autor:innen. Die Arbeit wurde bisher 22-mal zitiert (Crossref).
Abstract (Originalsprache)
High-resolution Vision-Language Models (VLMs) are widely used in multimodal tasks to enhance accuracy by preserving detailed image information. However, these models often generate an excessive number of visual tokens due to the need to encode multiple partitions of a high-resolution image input. Processing such a large number of visual tokens poses significant computational challenges, particularly for resource-constrained commodity GPUs. To address this challenge, we propose High-Resolution Early Dropping (HiRED), a plug-and-play token-dropping method designed to operate within a fixed token budget. HiRED leverages the attention of CLS token in the vision transformer (ViT) to assess the visual content of the image partitions and allocate an optimal token budget for each partition accordingly. The most informative visual tokens from each partition within the allocated budget are then selected and passed to the subsequent Large Language Model (LLM). We showed that HiRED achieves superior accuracy and performance, compared to existing token-dropping methods. Empirically, HiRED-20% (i.e., a 20% token budget) on LLaVA-Next-7B achieves a 4.7x increase in token generation throughput, reduces response latency by 78%, and saves 14% of GPU memory for single inference on an NVIDIA TESLA P40 (24 GB). For larger batch sizes (e.g., 4), HiRED-20% prevents out-of-memory errors by cutting memory usage by 30%, while preserving throughput and latency benefits.
Zitieren
APA
Arif, K. H. I., Yoon, J., Nikolopoulos, D. S., Vandierendonck, H., John, D., & Ji, B. (2025). HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 39(2), 1773-1781. https://doi.org/10.1609/aaai.v39i2.32171
Vancouver
Arif KHI, Yoon J, Nikolopoulos DS, Vandierendonck H, John D, Ji B. HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(2):1773-1781. doi:10.1609/aaai.v39i2.32171
BibTeX
@article{Arif2025,
author = {Arif, Kazi Hasan Ibn and Yoon, JinYi and Nikolopoulos, Dimitrios S. and Vandierendonck, Hans and John, Deepu and Ji, Bo},
title = {HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2025},
volume = {39},
number = {2},
pages = {1773--1781},
doi = {10.1609/aaai.v39i2.32171}
}
EUVIMED