EUVIMEDEUVIMED entdecken
Crossref

ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization

Hao Wang, Fang Liu, Licheng Jiao, Jiahao Wang, Zehua Hao, Shuo Li, Lingling Li, Puhua Chen, Xu Liu

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (6) · 5390-5400

DOI: 10.1609/aaai.v38i6.28347 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Hao Wang und 8 weiteren Autor:innen. Die Arbeit wurde bisher 52-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Pre-trained vision-language(V-L) models such as CLIP have demonstrated impressive Zero-Shot performance in many downstream tasks. Since adopting contrastive video-text pairs methods like CLIP to video tasks is limited by its high cost and scale, recent approaches focus on efficiently transferring the image-based CLIP to the video domain. A major finding is that fine-tuning the pre-trained model to achieve strong fully supervised performance leads to low zero shot, few shot, and base to novel generalization. Instead, freezing the backbone network to maintain generalization ability weakens fully supervised performance. Otherwise, no single prompt tuning branch consistently performs optimally. In this work, we proposed a multimodal prompt learning scheme that balances supervised and generalized performance. Our prompting approach contains three sections: 1) Independent prompt on both the vision and text branches to learn the language and visual contexts. 2) Inter-modal prompt mapping to ensure mutual synergy. 3) Reducing the discrepancy between the hand-crafted prompt (a video of a person doing [CLS]) and the learnable prompt, to alleviate the forgetting about essential video scenarios. Extensive validation of fully supervised, zero-shot, few-shot, base-to-novel generalization settings for video recognition indicates that the proposed approach achieves competitive performance with less commute cost.

Zitieren

APA

Wang, H., Liu, F., Jiao, L., Wang, J., Hao, Z., Li, S., Li, L., Chen, P., & Liu, X. (2024). ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6), 5390-5400. https://doi.org/10.1609/aaai.v38i6.28347

Vancouver

Wang H, Liu F, Jiao L, Wang J, Hao Z, Li S, et al. ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(6):5390-5400. doi:10.1609/aaai.v38i6.28347

BibTeX
@article{Wang2024,
  author = {Wang, Hao and Liu, Fang and Jiao, Licheng and Wang, Jiahao and Hao, Zehua and Li, Shuo and Li, Lingling and Chen, Puhua and Liu, Xu},
  title = {ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {6},
  pages = {5390--5400},
  doi = {10.1609/aaai.v38i6.28347}
}
Kostenlos starten →