From CNNs to Transformers in Multimodal Human Action Recognition: A Survey
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.08.2024 · 20 (8) · 1-24
DOI: 10.1145/3664815 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 09.07.2024. Verfasst von Muhammad Bilal Shaikh und 3 weiteren Autor:innen. Die Arbeit wurde bisher 67-mal zitiert (Crossref) und nennt 161 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-SA 4.0). Gefördert von: Edith Cowan University (ECU) and the Higher Education Commission (HEC) of Pakistan, Office of National Intelligence National Intelligence Postdoctoral.
Abstract (Originalsprache)
Due to its widespread applications, human action recognition is one of the most widely studied research problems in Computer Vision. Recent studies have shown that addressing it using multimodal data leads to superior performance as compared to relying on a single data modality. During the adoption of deep learning for visual modelling in the past decade, action recognition approaches have mainly relied on Convolutional Neural Networks (CNNs). However, the recent rise of Transformers in visual modelling is now also causing a paradigm shift for the action recognition task. This survey captures this transition while focusing on Multimodal Human Action Recognition (MHAR). Unique to the induction of multimodal computational models is the process of ‘fusing’ the features of the individual data modalities. Hence, we specifically focus on the fusion design aspects of the MHAR approaches. We analyze the classic and emerging techniques in this regard, while also highlighting the popular trends in the adaption of CNN and Transformer building blocks for the overall problem. In particular, we emphasize on recent design choices that have led to more efficient MHAR models. Unlike existing reviews, which discuss Human Action Recognition from a broad perspective, this survey is specifically aimed at pushing the boundaries of MHAR research by identifying promising architectural and fusion design choices to train practicable models. We also provide an outlook of the multimodal datasets from their scale and evaluation viewpoint. Finally, building on the reviewed literature, we discuss the challenges and future avenues for MHAR.
Förderung
- Edith Cowan University (ECU) and the Higher Education Commission (HEC) of Pakistan
- Office of National Intelligence National Intelligence Postdoctoral
Zitieren
APA
Shaikh, M. B., Chai, D., Islam, S. M. S., & Akhtar, N. (2024). From CNNs to Transformers in Multimodal Human Action Recognition: A Survey. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(8), 1-24. https://doi.org/10.1145/3664815
Vancouver
Shaikh MB, Chai D, Islam SMS, Akhtar N. From CNNs to Transformers in Multimodal Human Action Recognition: A Survey. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(8):1-24. doi:10.1145/3664815
BibTeX
@article{Shaikh2024,
author = {Shaikh, Muhammad Bilal and Chai, Douglas and Islam, Syed Muhammad Shamsul and Akhtar, Naveed},
title = {From CNNs to Transformers in Multimodal Human Action Recognition: A Survey},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2024},
volume = {20},
number = {8},
pages = {1--24},
doi = {10.1145/3664815}
}
EUVIMED