Diversified Composite Prompting to Enhance Generalisation of Vision‐Language Models
CAAI Transactions on Intelligence Technology · 2026-06 · 11 (3) · 935-950
DOI: 10.1049/cit2.70143 ↗Kurz gefasst
Fachartikel in der Zeitschrift CAAI Transactions on Intelligence Technology, online erschienen am 13.05.2026. Verfasst von Xiaoyong Mei und 5 weiteren Autor:innen. Die Arbeit nennt 68 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Key Research and Development Program of China, Zhejiang Office of Philosophy and Social Science.
Abstract (Originalsprache)
In recent years, prompt learning has shown promise in transferring pretrained vision‐language models (VLMs) to downstream tasks. However, existing methods face two challenges in improving generalisation: (1) When leveraging the collaborative effect of multimodal prompts, it is often assumed that text and visual modalities share the same prompt requirements, neglecting the distinct hierarchical processing of their encoders, leading to prompt imbalance; and (2) current methods exhibit limited adaptability when facing diverse distribution shift scenarios, including class distribution shifts and image content variations. To address these challenges, we propose a diversified composite prompt learning (DCPL) framework that integrates unified and specific prompts. Specifically, to alleviate multimodal prompt imbalance, we design a shared root multimodal prompting strategy, which employs a shared root prompt and an independent derivation mechanism to generate the derived multimodal prompt (DMP), enabling independent deep prompting while maintaining implicit synergy across modalities. Furthermore, we design a dual‐branch dynamic adaptive prompting strategy that produces the derived class‐specific prompt (DCP) and image‐specific prompt (ISP), driven by inter‐class relations and image‐patch context, respectively, to enhance adaptability across different distribution shifts. Extensive experiments on base‐to‐novel, cross‐dataset, domain generalisation and few‐shot learning demonstrate that the DCPL achieves superior performance, validating its robustness and generalisation.
Förderung
- National Key Research and Development Program of China
- Zhejiang Office of Philosophy and Social Science
Zitieren
APA
Mei, X., Tang, C., Dai, Z., Zheng, F., Zhang, K., & Lin, T. (2026). Diversified Composite Prompting to Enhance Generalisation of Vision‐Language Models. CAAI Transactions on Intelligence Technology, 11(3), 935-950. https://doi.org/10.1049/cit2.70143
Vancouver
Mei X, Tang C, Dai Z, Zheng F, Zhang K, Lin T. Diversified Composite Prompting to Enhance Generalisation of Vision‐Language Models. CAAI Transactions on Intelligence Technology. 2026;11(3):935-950. doi:10.1049/cit2.70143
BibTeX
@article{Mei2026,
author = {Mei, Xiaoyong and Tang, Chong and Dai, Zhengqun and Zheng, Fudan and Zhang, Kongwen and Lin, Tianyu},
title = {Diversified Composite Prompting to Enhance Generalisation of Vision‐Language Models},
journal = {CAAI Transactions on Intelligence Technology},
year = {2026},
volume = {11},
number = {3},
pages = {935--950},
doi = {10.1049/cit2.70143}
}
EUVIMED