AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning
Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (3) · 3077-3085
DOI: 10.1609/aaai.v39i3.32316 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Jun Gao und 5 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref).
Abstract (Originalsprache)
In-context learning (ICL) advances Large Language Models (LLMs) exhibiting emergent ability on downstream tasks without updating billions of parameters. However, in the area of multimodal Large Language Models (MLLMs), two problems hinder the application of multimodal ICL: (1) Most primary MLLMs are only trained on single-image datasets, making them unable to read extra multimodal demonstrations. (2) With the demonstrations increasing, thousands of visual tokens highly challenge hardware and degrade ICL performance. During preliminary explorations, we discovered that the inner LLM focuses more on the linguistic modality within multimodal demonstrations during generation. Therefore, we propose a general and lightweight framework AIM to tackle the mentioned problems through Aggregating Image information of Multimodal demonstrations to the latent space of the corresponding textual labels. After aggregation, AIM substitutes each demonstration with generated fused virtual tokens whose length is reduced to the same as its texts. Except for shortening input length, AIM further upgrades MLLMs pre-trained on image-text pairs to support multimodal ICL, as images from demonstrations are disregarded. Furthermore, benefiting from aggregating different demonstrations independently, AIM configures Demonstration Bank (DB) to avoid repeated aggregation, which significantly boosts model efficiency. We build AIM upon QWen-VL and LLaVA-Next, and AIM is comprehensively evaluated on image caption, VQA, and hateful speech detection. Outstanding results reveal that AIM provides an efficient and effective solution in upgrading MLLMs for multimodal ICL.
Zitieren
APA
Gao, J., Qiao, Q., Wu, T., Wang, Z., Cao, Z., & Li, W. (2025). AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 39(3), 3077-3085. https://doi.org/10.1609/aaai.v39i3.32316
Vancouver
Gao J, Qiao Q, Wu T, Wang Z, Cao Z, Li W. AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(3):3077-3085. doi:10.1609/aaai.v39i3.32316
BibTeX
@article{Gao2025,
author = {Gao, Jun and Qiao, Qian and Wu, Tianxiang and Wang, Zili and Cao, Ziqiang and Li, Wenjie},
title = {AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2025},
volume = {39},
number = {3},
pages = {3077--3085},
doi = {10.1609/aaai.v39i3.32316}
}
EUVIMED