EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Diversified Composite Prompting to Enhance Generalisation of Vision‐Language Models

Xiaoyong Mei, Chong Tang, Zhengqun Dai, Fudan Zheng, Kongwen Zhang, Tianyu Lin

CAAI Transactions on Intelligence Technology · 2026-06 · 11 (3) · 935-950

DOI: 10.1049/cit2.70143 ↗

Kurz gefasst

Fachartikel in der Zeitschrift CAAI Transactions on Intelligence Technology, online erschienen am 13.05.2026. Verfasst von Xiaoyong Mei und 5 weiteren Autor:innen. Die Arbeit nennt 68 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Key Research and Development Program of China, Zhejiang Office of Philosophy and Social Science.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In recent years, prompt learning has shown promise in transferring pretrained vision‐language models (VLMs) to downstream tasks. However, existing methods face two challenges in improving generalisation: (1) When leveraging the collaborative effect of multimodal prompts, it is often assumed that text and visual modalities share the same prompt requirements, neglecting the distinct hierarchical processing of their encoders, leading to prompt imbalance; and (2) current methods exhibit limited adaptability when facing diverse distribution shift scenarios, including class distribution shifts and image content variations. To address these challenges, we propose a diversified composite prompt learning (DCPL) framework that integrates unified and specific prompts. Specifically, to alleviate multimodal prompt imbalance, we design a shared root multimodal prompting strategy, which employs a shared root prompt and an independent derivation mechanism to generate the derived multimodal prompt (DMP), enabling independent deep prompting while maintaining implicit synergy across modalities. Furthermore, we design a dual‐branch dynamic adaptive prompting strategy that produces the derived class‐specific prompt (DCP) and image‐specific prompt (ISP), driven by inter‐class relations and image‐patch context, respectively, to enhance adaptability across different distribution shifts. Extensive experiments on base‐to‐novel, cross‐dataset, domain generalisation and few‐shot learning demonstrate that the DCPL achieves superior performance, validating its robustness and generalisation.

Förderung

  • National Key Research and Development Program of China
  • Zhejiang Office of Philosophy and Social Science

Zitieren

APA

Mei, X., Tang, C., Dai, Z., Zheng, F., Zhang, K., & Lin, T. (2026). Diversified Composite Prompting to Enhance Generalisation of Vision‐Language Models. CAAI Transactions on Intelligence Technology, 11(3), 935-950. https://doi.org/10.1049/cit2.70143

Vancouver

Mei X, Tang C, Dai Z, Zheng F, Zhang K, Lin T. Diversified Composite Prompting to Enhance Generalisation of Vision‐Language Models. CAAI Transactions on Intelligence Technology. 2026;11(3):935-950. doi:10.1049/cit2.70143

BibTeX
@article{Mei2026,
  author = {Mei, Xiaoyong and Tang, Chong and Dai, Zhengqun and Zheng, Fudan and Zhang, Kongwen and Lin, Tianyu},
  title = {Diversified Composite Prompting to Enhance Generalisation of Vision‐Language Models},
  journal = {CAAI Transactions on Intelligence Technology},
  year = {2026},
  volume = {11},
  number = {3},
  pages = {935--950},
  doi = {10.1049/cit2.70143}
}
Kostenlos starten →