EUVIMEDEUVIMED entdecken
Crossref

MAR: M asked Autoencoders for Efficient A ction R ecognition

Zhiwu Qing, Shiwei Zhang, Ziyuan Huang, Xiang Wang, Yuehuan Wang, Yiliang Lv, Changxin Gao, Nong Sang

IEEE Transactions on Multimedia · 2024 · 26 · 218-233

DOI: 10.1109/tmm.2023.3263288 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Multimedia, 2024. Verfasst von Zhiwu Qing und 7 weiteren Autor:innen. Die Arbeit wurde bisher 58-mal zitiert (Crossref) und nennt 100 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Alibaba DAMO Academy.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Standard approaches for video action recognition usually operate on full input videos, which is inefficient due to the widespread spatio-temporal redundancy in videos. The recent progress in masked video modelling, specifically VideoMAE, has shown the ability of vanilla Vision Transformers (ViT) to complement spatio-temporal contexts using limited visual content. Inspired by this, we propose Masked Action Recognition (MAR), which reduces redundant computation by discarding a proportion of patches and operating only on a portion of the videos. MAR includes two essential components: cell running masking and bridging classifier. Specifically, to enable the ViT to perceive the details beyond the visible patches, cell running masking is used to preserve the spatio-temporal correlations in videos. This ensures that the patches at the same spatial location can be observed in turn for easy reconstructions. Additionally, we notice that, although the partially observed features can reconstruct semantically explicit invisible patches, they fail to achieve accurate classification. To address this issue, we propose a bridging classifier that can help fill the semantic gap between the ViT encoded features used for reconstruction and the specialized features used for classification. Our proposed MAR can reduce the computational cost of ViT by 53%. Extensive experiments have demonstrated that MAR consistently outperforms existing ViT models by a notable margin. Notably, we found that a ViT-Large model fine-tuned by MAR achieves comparable performance to a ViT-Huge model fine-tuned by standard training methods on both Kinetics-400 and Something-Something v2 datasets. Moreover, the computation overhead of our ViT-Large model is only 14.5% of that of the ViT-Huge model.

Förderung

  • National Natural Science Foundation of China
  • Alibaba DAMO Academy

Zitieren

APA

Qing, Z., Zhang, S., Huang, Z., Wang, X., Wang, Y., Lv, Y., Gao, C., & Sang, N. (2024). MAR: M asked Autoencoders for Efficient A ction R ecognition. IEEE Transactions on Multimedia, 26, 218-233. https://doi.org/10.1109/tmm.2023.3263288

Vancouver

Qing Z, Zhang S, Huang Z, Wang X, Wang Y, Lv Y, et al. MAR: M asked Autoencoders for Efficient A ction R ecognition. IEEE Transactions on Multimedia. 2024;26:218-233. doi:10.1109/tmm.2023.3263288

BibTeX
@article{Qing2024,
  author = {Qing, Zhiwu and Zhang, Shiwei and Huang, Ziyuan and Wang, Xiang and Wang, Yuehuan and Lv, Yiliang and Gao, Changxin and Sang, Nong},
  title = {MAR: M asked Autoencoders for Efficient A ction R ecognition},
  journal = {IEEE Transactions on Multimedia},
  year = {2024},
  volume = {26},
  pages = {218--233},
  doi = {10.1109/tmm.2023.3263288}
}
Kostenlos starten →