Multi-Modality Adaptive Feature Fusion Graph Convolutional Network for Skeleton-Based Action Recognition
Sensors · 07.06.2023 · 23 (12) · 5414
DOI: 10.3390/s23125414 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 07.06.2023. Verfasst von Haiping Zhang und 6 weiteren Autor:innen. Beteiligte Einrichtung: School of Computer Science, Hangzhou Dianzi University, Hangzhou 310005, China. Hauptthemen laut MEDLINE: Musculoskeletal System. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Research and Application of edge computing Technology Based on TinyML.
Abstract (Originalsprache)
Graph convolutional networks are widely used in skeleton-based action recognition because of their good fitting ability to non-Euclidean data. While conventional multi-scale temporal convolution uses several fixed-size convolution kernels or dilation rates at each layer of the network, we argue that different layers and datasets require different receptive fields. We use multi-scale adaptive convolution kernels and dilation rates to optimize traditional multi-scale temporal convolution with a simple and effective self attention mechanism, allowing different network layers to adaptively select convolution kernels of different sizes and dilation rates instead of being fixed and unchanged. Besides, the effective receptive field of the simple residual connection is not large, and there is a great deal of redundancy in the deep residual network, which will lead to the loss of context when aggregating spatio-temporal information. This article introduces a feature fusion mechanism that replaces the residual connection between initial features and temporal module outputs, effectively solving the problems of context aggregation and initial feature fusion. We propose a multi-modality adaptive feature fusion framework (MMAFF) to simultaneously increase the receptive field in both spatial and temporal dimensions. Concretely, we input the features extracted by the spatial module into the adaptive temporal fusion module to simultaneously extract multi-scale skeleton features in both spatial and temporal parts. In addition, based on the current multi-stream approach, we use the limb stream to uniformly process correlated data from multiple modalities. Extensive experiments show that our model obtains competitive results with state-of-the-art methods on the NTU-RGB+D 60 and NTU-RGB+D 120 datasets.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Computer Science, Hangzhou Dianzi University, Hangzhou 310005, China.
- School of Information Engineering, Hangzhou Dianzi University, Hangzhou 310005, China.
- School of Electronics and Information, Hangzhou Dianzi University, Hangzhou 310005, China.
ORCID-Profile: Xinhao Zhang, Dongjing Wang, Wanjun Zhang
Förderung
- Research and Application of edge computing Technology Based on TinyML
Zitieren
APA
Zhang, H., Zhang, X., Yu, D., Guan, L., Wang, D., Zhou, F., & Zhang, W. (2023). Multi-Modality Adaptive Feature Fusion Graph Convolutional Network for Skeleton-Based Action Recognition. Sensors, 23(12), 5414. https://doi.org/10.3390/s23125414
Vancouver
Zhang H, Zhang X, Yu D, Guan L, Wang D, Zhou F, et al. Multi-Modality Adaptive Feature Fusion Graph Convolutional Network for Skeleton-Based Action Recognition. Sensors. 2023;23(12):5414. doi:10.3390/s23125414
BibTeX
@article{Zhang2023,
author = {Zhang, Haiping and Zhang, Xinhao and Yu, Dongjin and Guan, Liming and Wang, Dongjing and Zhou, Fuxing and Zhang, Wanjun},
title = {Multi-Modality Adaptive Feature Fusion Graph Convolutional Network for Skeleton-Based Action Recognition},
journal = {Sensors},
year = {2023},
volume = {23},
number = {12},
pages = {5414},
doi = {10.3390/s23125414}
}
EUVIMED