RMAdapter: Reconstruction-based Multi-Modal Adapter for Vision-Language Models
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (28) · 23594-23602
DOI: 10.1609/aaai.v40i28.39532 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Xiang Lin und 4 weiteren Autor:innen.
Abstract (Originalsprache)
Pre-trained Vision-Language Models (VLMs), e.g. CLIP, have become essential tools in multimodal transfer learning. However, fine-tuning VLMs in few-shot scenarios poses significant challenges in balancing task-specific adaptation and generalization in the obtained model. Meanwhile, current researches have predominantly focused on prompt-based adaptation methods, leaving adapter-based approaches underexplored and revealing notable performance gaps. To address these challenges, we introduce a novel Reconstruction-based Multimodal Adapter (RMAdapter), which leverages a dual-branch architecture. Unlike conventional single-branch adapters, RMAdapter consists of: (1) an adaptation branch that injects task-specific knowledge through parameter-efficient fine-tuning, and (2) a reconstruction branch that preserves general knowledge by reconstructing latent space features back into the original feature space. This design facilitates a dynamic balance between general and task-specific knowledge. Importantly, although RMAdapter introduces an additional reconstruction branch, it is carefully optimized to remain lightweight. By computing reconstruction loss locally at each layer and sharing projection modules, the overall computational overhead is kept minimal. A consistency constraint is also incorporated to better regulate the trade-off between discriminability and generalization. We comprehensively evaluate the effectiveness of RMAdapter on three representative tasks: generalization to new categories, generalization to new target datasets, and domain generalization. Without relying on data augmentation or duplicate prompt designs, our RMAdapter consistently outperforms state-of-the-art approaches across all evaluation metrics.
Zitieren
APA
Lin, X., Li, W., Guo, S., Wang, L., & Huang, D. (2026). RMAdapter: Reconstruction-based Multi-Modal Adapter for Vision-Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 40(28), 23594-23602. https://doi.org/10.1609/aaai.v40i28.39532
Vancouver
Lin X, Li W, Guo S, Wang L, Huang D. RMAdapter: Reconstruction-based Multi-Modal Adapter for Vision-Language Models. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(28):23594-23602. doi:10.1609/aaai.v40i28.39532
BibTeX
@article{Lin2026,
author = {Lin, Xiang and Li, Weixin and Guo, Shu and Wang, Lihong and Huang, Di},
title = {RMAdapter: Reconstruction-based Multi-Modal Adapter for Vision-Language Models},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {28},
pages = {23594--23602},
doi = {10.1609/aaai.v40i28.39532}
}
EUVIMED