EUVIMEDEUVIMED entdecken
Crossref

Leveraging Large Vision-Language Model as User Intent-Aware Encoder for Composed Image Retrieval

Zelong Sun, Dong Jing, Guoxing Yang, Nanyi Fei, Zhiwu Lu

Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (7) · 7149-7157

DOI: 10.1609/aaai.v39i7.32768 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Zelong Sun und 4 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Composed Image Retrieval (CIR) aims to retrieve target images from candidate set using a hybrid-modality query consisting of a reference image and a relative caption that describes the user intent. Recent studies attempt to utilize Vision-Language Pre-training Models (VLPMs) with various fusion strategies for addressing the task. However, these methods typically fail to simultaneously meet two key requirements of CIR: comprehensively extracting visual information and faithfully following the user intent. In this work, we propose CIR-LVLM, a novel framework that leverages the large vision-language model (LVLM) as the powerful user intent-aware encoder to better meet these requirements. Our motivation is to explore the advanced reasoning and instruction-following capabilities of LVLM for accurately understanding and responding the user intent. Furthermore, we design a novel hybrid intent instruction module to provide explicit intent guidance at two levels: (1) The task prompt clarifies the task requirement and assists the model in discerning user intent at the task level. (2) The instance-specific soft prompt, which is adaptively selected from the learnable prompt pool, enables the model to better comprehend the user intent at the instance level compared to a universal prompt for all instances. CIR-LVLM achieves state-of-the-art performance across three prominent benchmarks with acceptable inference efficiency. We believe this study provides fundamental insights into CIR-related fields.

Zitieren

APA

Sun, Z., Jing, D., Yang, G., Fei, N., & Lu, Z. (2025). Leveraging Large Vision-Language Model as User Intent-Aware Encoder for Composed Image Retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, 39(7), 7149-7157. https://doi.org/10.1609/aaai.v39i7.32768

Vancouver

Sun Z, Jing D, Yang G, Fei N, Lu Z. Leveraging Large Vision-Language Model as User Intent-Aware Encoder for Composed Image Retrieval. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(7):7149-7157. doi:10.1609/aaai.v39i7.32768

BibTeX
@article{Sun2025,
  author = {Sun, Zelong and Jing, Dong and Yang, Guoxing and Fei, Nanyi and Lu, Zhiwu},
  title = {Leveraging Large Vision-Language Model as User Intent-Aware Encoder for Composed Image Retrieval},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2025},
  volume = {39},
  number = {7},
  pages = {7149--7157},
  doi = {10.1609/aaai.v39i7.32768}
}
Kostenlos starten →