DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (7) · 6540-6548
DOI: 10.1609/aaai.v38i7.28475 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Xiangpeng Yang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 40-mal zitiert (Crossref).
Abstract (Originalsprache)
Text-video retrieval is a critical multi-modal task to find the most relevant video for a text query. Although pretrained models like CLIP have demonstrated impressive potential in this area, the rising cost of fully finetuning these models due to increasing model size continues to pose a problem. To address this challenge, prompt tuning has emerged as an alternative. However, existing works still face two problems when adapting pretrained image-text models to downstream video-text tasks: (1) The visual encoder could only encode frame-level features and failed to extract global-level general video information. (2) Equipping the visual and text encoder with separated prompts failed to mitigate the visual-text modality gap. To this end, we propose DGL, a cross-modal Dynamic prompt tuning method with Global-Local video attention. In contrast to previous prompt tuning methods, we employ the shared latent space to generate local-level text and frame prompts that encourage inter-modal interaction. Furthermore, we propose modeling video in a global-local attention mechanism to capture global video information from the perspective of prompt tuning. Extensive experiments reveal that when only 0.67% parameters are tuned, our cross-modal prompt tuning strategy DGL outperforms or is comparable to fully finetuning methods on MSR-VTT, VATEX, LSMDC, and ActivityNet datasets. Code will be available at https://github.com/knightyxp/DGL.
Zitieren
APA
Yang, X., Zhu, L., Wang, X., & Yang, Y. (2024). DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, 38(7), 6540-6548. https://doi.org/10.1609/aaai.v38i7.28475
Vancouver
Yang X, Zhu L, Wang X, Yang Y. DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(7):6540-6548. doi:10.1609/aaai.v38i7.28475
BibTeX
@article{Yang2024,
author = {Yang, Xiangpeng and Zhu, Linchao and Wang, Xiaohan and Yang, Yi},
title = {DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {7},
pages = {6540--6548},
doi = {10.1609/aaai.v38i7.28475}
}
EUVIMED