EUVIMEDEUVIMED entdecken
Crossref

A Multimodal, Multi-Task Adapting Framework for Video Action Recognition

Mengmeng Wang, Jiazheng Xing, Boyuan Jiang, Jun Chen, Jianbiao Mei, Xingxing Zuo, Guang Dai, Jingdong Wang, Yong Liu

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (6) · 5517-5525

DOI: 10.1609/aaai.v38i6.28361 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Mengmeng Wang und 8 weiteren Autor:innen. Die Arbeit wurde bisher 45-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recently, the rise of large-scale vision-language pretrained models like CLIP, coupled with the technology of Parameter-Efficient FineTuning (PEFT), has captured substantial attraction in video action recognition. Nevertheless, prevailing approaches tend to prioritize strong supervised performance at the expense of compromising the models' generalization capabilities during transfer. In this paper, we introduce a novel Multimodal, Multi-task CLIP adapting framework named M2-CLIP to address these challenges, preserving both high supervised performance and robust transferability. Firstly, to enhance the individual modality architectures, we introduce multimodal adapters to both the visual and text branches. Specifically, we design a novel visual TED-Adapter, that performs global Temporal Enhancement and local temporal Difference modeling to improve the temporal representation capabilities of the visual encoder. Moreover, we adopt text encoder adapters to strengthen the learning of semantic label information. Secondly, we design a multi-task decoder with a rich set of supervisory signals, including the original contrastive learning head, a cross-modal classification head, a cross-modal masked language modeling head, and a visual classification head. This multi-task decoder adeptly satisfies the need for strong supervised performance within a multimodal framework. Experimental results validate the efficacy of our approach, demonstrating exceptional performance in supervised learning while maintaining strong generalization in zero-shot scenarios.

Zitieren

APA

Wang, M., Xing, J., Jiang, B., Chen, J., Mei, J., Zuo, X., Dai, G., Wang, J., & Liu, Y. (2024). A Multimodal, Multi-Task Adapting Framework for Video Action Recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6), 5517-5525. https://doi.org/10.1609/aaai.v38i6.28361

Vancouver

Wang M, Xing J, Jiang B, Chen J, Mei J, Zuo X, et al. A Multimodal, Multi-Task Adapting Framework for Video Action Recognition. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(6):5517-5525. doi:10.1609/aaai.v38i6.28361

BibTeX
@article{Wang2024,
  author = {Wang, Mengmeng and Xing, Jiazheng and Jiang, Boyuan and Chen, Jun and Mei, Jianbiao and Zuo, Xingxing and Dai, Guang and Wang, Jingdong and Liu, Yong},
  title = {A Multimodal, Multi-Task Adapting Framework for Video Action Recognition},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {6},
  pages = {5517--5525},
  doi = {10.1609/aaai.v38i6.28361}
}
Kostenlos starten →