ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (6) · 5390-5400
DOI: 10.1609/aaai.v38i6.28347 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Hao Wang und 8 weiteren Autor:innen. Die Arbeit wurde bisher 52-mal zitiert (Crossref).
Abstract (Originalsprache)
Pre-trained vision-language(V-L) models such as CLIP have demonstrated impressive Zero-Shot performance in many downstream tasks. Since adopting contrastive video-text pairs methods like CLIP to video tasks is limited by its high cost and scale, recent approaches focus on efficiently transferring the image-based CLIP to the video domain. A major finding is that fine-tuning the pre-trained model to achieve strong fully supervised performance leads to low zero shot, few shot, and base to novel generalization. Instead, freezing the backbone network to maintain generalization ability weakens fully supervised performance. Otherwise, no single prompt tuning branch consistently performs optimally. In this work, we proposed a multimodal prompt learning scheme that balances supervised and generalized performance. Our prompting approach contains three sections: 1) Independent prompt on both the vision and text branches to learn the language and visual contexts. 2) Inter-modal prompt mapping to ensure mutual synergy. 3) Reducing the discrepancy between the hand-crafted prompt (a video of a person doing [CLS]) and the learnable prompt, to alleviate the forgetting about essential video scenarios. Extensive validation of fully supervised, zero-shot, few-shot, base-to-novel generalization settings for video recognition indicates that the proposed approach achieves competitive performance with less commute cost.
Zitieren
APA
Wang, H., Liu, F., Jiao, L., Wang, J., Hao, Z., Li, S., Li, L., Chen, P., & Liu, X. (2024). ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6), 5390-5400. https://doi.org/10.1609/aaai.v38i6.28347
Vancouver
Wang H, Liu F, Jiao L, Wang J, Hao Z, Li S, et al. ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(6):5390-5400. doi:10.1609/aaai.v38i6.28347
BibTeX
@article{Wang2024,
author = {Wang, Hao and Liu, Fang and Jiao, Licheng and Wang, Jiahao and Hao, Zehua and Li, Shuo and Li, Lingling and Chen, Puhua and Liu, Xu},
title = {ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {6},
pages = {5390--5400},
doi = {10.1609/aaai.v38i6.28347}
}
EUVIMED