EUVIMEDEUVIMED entdecken
Crossref

SMTDKD: A Semantic-Aware Multimodal Transformer Fusion Decoupled Knowledge Distillation Method for Action Recognition

Zhenzhen Quan, Qingshan Chen, Wei Wang, Moyan Zhang, Xiang Li, Yujun Li, Zhi Liu

IEEE Sensors Journal · 15.01.2024 · 24 (2) · 2289-2304

DOI: 10.1109/jsen.2023.3337367 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Sensors Journal, erschienen am 15.01.2024. Verfasst von Zhenzhen Quan und 6 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 81 Literaturangaben. Gefördert von: Qingdao Municipal Science and Technology Benefit People Demonstration and Guidance Special Project, Major Scientific and Technological Innovation Project in Shandong Province, Natural Science Foundation of Shandong Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal sensors, including vision sensors and wearable sensors, offer valuable complementary information for accurate recognition tasks. Nonetheless, the heterogeneity among sensor data from different modalities presents a formidable challenge in extracting robust multimodal information amidst noise. In this article, we propose an innovative approach, named semantic-aware multimodal transformer fusion decoupled knowledge distillation (SMTDKD) method, which guides video data recognition not only through the information interaction between different wearable-sensor data, but also through the information interaction between visual sensor data and wearable-sensor data, improving the robustness of the model. To preserve the temporal relationship within wearable-sensor data, the SMTDKD method converts them into 2-D image data. Furthermore, a transformer-based multimodal fusion module is designed to capture diverse feature information from distinct wearable-sensor modalities. To mitigate modality discrepancies and encourage similar semantic features, graph cross-view attention maps are constructed across various convolutional layers to facilitate feature alignment. Additionally, semantic information is exchanged among the teacher–student network, the student network, and bidirectional encoder representations from transformer (BERT)-encoded labels. To obtain more comprehensive knowledge transfer, the decoupled knowledge distillation loss is utilized, thus enhancing the generalization of the network. Experimental evaluations conducted on three multimodal datasets, namely, UTD-MHAD, Berkeley-MHAD, and MMAct, demonstrate the superior performance of the proposed SMTDKD method over the state-of-the-art action human recognition methods.

Förderung

  • Qingdao Municipal Science and Technology Benefit People Demonstration and Guidance Special Project
  • Major Scientific and Technological Innovation Project in Shandong Province
  • Natural Science Foundation of Shandong Province

Zitieren

APA

Quan, Z., Chen, Q., Wang, W., Zhang, M., Li, X., Li, Y., & Liu, Z. (2024). SMTDKD: A Semantic-Aware Multimodal Transformer Fusion Decoupled Knowledge Distillation Method for Action Recognition. IEEE Sensors Journal, 24(2), 2289-2304. https://doi.org/10.1109/jsen.2023.3337367

Vancouver

Quan Z, Chen Q, Wang W, Zhang M, Li X, Li Y, et al. SMTDKD: A Semantic-Aware Multimodal Transformer Fusion Decoupled Knowledge Distillation Method for Action Recognition. IEEE Sensors Journal. 2024;24(2):2289-2304. doi:10.1109/jsen.2023.3337367

BibTeX
@article{Quan2024,
  author = {Quan, Zhenzhen and Chen, Qingshan and Wang, Wei and Zhang, Moyan and Li, Xiang and Li, Yujun and Liu, Zhi},
  title = {SMTDKD: A Semantic-Aware Multimodal Transformer Fusion Decoupled Knowledge Distillation Method for Action Recognition},
  journal = {IEEE Sensors Journal},
  year = {2024},
  volume = {24},
  number = {2},
  pages = {2289--2304},
  doi = {10.1109/jsen.2023.3337367}
}
Kostenlos starten →