Leveraging Image as Compressed Visual Prompt and Hierarchical Visual Knowledge for Effective Image Utilization in MLLMs
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (30) · 25554-25562
DOI: 10.1609/aaai.v40i30.39751 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Shezheng Song und 8 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).
Abstract (Originalsprache)
Multimodal Large Language Models (MLLMs) integrate text and images for complex reasoning tasks, but efficiently utilizing image remains a challenge due to redundancy and noise. Traditional methods take the entire image features as visual prompt into the MLLMs, leading to excessive visual tokens that disrupt textual information expression. Thus, recent studies treat image features as visual knowledge, storing them in the feed-forward network for retrieval when needed. These methods, completely removing images from the input, may hinder the activation of image-related knowledge. Besides, current visual knowledge focuses on fine-grained details but overlooks the hierarchical process of visual perception. As described in feature integration theory, global structure is first processed before details are integrated. Ignoring this process may lead to a fragmented visual understanding, making it difficult to capture high-level semantic relationships. To overcome these issues, we propose a novel image utilization mechanism in MLLMs. We leverage a compression-based attention mechanism to generate the compressed visual prompt, which not only mitigates the interference of excessively long visual prompts but also preserves crucial visual information necessary for activating knowledge in the MLLM. Furthermore, we extract hierarchical visual features as visual knowledge using wavelet transforms, allowing the model to capture both global structures and fine-grained details. Experiments show that our method achieves state-of-the-art performance.
Zitieren
APA
Song, S., Ding, K., Zhao, S., Li, S., Li, X., Wang, C., Wan, Q., Ji, B., & Yu, J. (2026). Leveraging Image as Compressed Visual Prompt and Hierarchical Visual Knowledge for Effective Image Utilization in MLLMs. Proceedings of the AAAI Conference on Artificial Intelligence, 40(30), 25554-25562. https://doi.org/10.1609/aaai.v40i30.39751
Vancouver
Song S, Ding K, Zhao S, Li S, Li X, Wang C, et al. Leveraging Image as Compressed Visual Prompt and Hierarchical Visual Knowledge for Effective Image Utilization in MLLMs. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(30):25554-25562. doi:10.1609/aaai.v40i30.39751
BibTeX
@article{Song2026,
author = {Song, Shezheng and Ding, Kangcheng and Zhao, Shan and Li, Shasha and Li, Xiaopeng and Wang, Chengyu and Wan, Qian and Ji, Bin and Yu, Jie},
title = {Leveraging Image as Compressed Visual Prompt and Hierarchical Visual Knowledge for Effective Image Utilization in MLLMs},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {30},
pages = {25554--25562},
doi = {10.1609/aaai.v40i30.39751}
}
EUVIMED