EUVIMEDEUVIMED entdecken
Crossref

R2G$^{3}$ Net: A Novel Hierarchical Spatial-Temporal Neural Network With a Regional-to-Global Fusion Mechanism for Multimodal Emotion Recognition

Yanan Zhang, Chenxu Guo, Kexin Zhu, Wenbo Hu, Bin Hu, Jian Shen

IEEE Transactions on Affective Computing · 2026-04 · 17 (2) · 1774-1787

DOI: 10.1109/taffc.2026.3672521 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, 2026-04. Verfasst von Yanan Zhang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref). Gefördert von: National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities, China Postdoctoral Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the rapid advancement of emotion recognition technology, multimodal physiological signals have garnered increasing research attention due to their rich affective representations. However, the substantial heterogeneity across different physiological modalities poses a significant challenge for effective multimodal fusion, limiting the performance of current emotion recognition systems. Moreover, while demographic information inherently encodes valuable emotional cues, its systematic integration into emotion recognition remains underexplored. To address these challenges, we propose R2G$^{3}$Net, a novel hierarchical framework for multimodal emotion recognition. Our model leverages a three-tier architecture: 1) Regional-to-Global Brain Feature Extraction: A BiLSTM-GNN hybrid network hierarchically encodes EEG signals, capturing spatio-temporal patterns from local brain regions to global functional connectivity. 2) Regional-to-Global Cross-Modal Fusion: Peripheral nervous system (PNS) signals are extracted and fused with brain features to enhance physiological representation learning. 3) Regional-to-Global Social Context-Aware Modeling: A hypergraph neural network (HGNN) integrates demographic data to construct dynamic social networks, uncovering higher-order emotional interactions for improved interpretability. Extensive experiments on three benchmark datasets demonstrate R2G$^{3}$Net's superiority in joint spatio-temporal feature learning and social context-aware emotion recognition. Ablation studies and visual analytics further validate that our fused representations outperform state-of-the-art methods in both discriminative capability and model transparency.

Förderung

  • National Natural Science Foundation of China
  • Fundamental Research Funds for the Central Universities
  • China Postdoctoral Science Foundation
  • China University Research Innovation Fund

Zitieren

APA

Zhang, Y., Guo, C., Zhu, K., Hu, W., Hu, B., & Shen, J. (2026). R2G$^{3}$ Net: A Novel Hierarchical Spatial-Temporal Neural Network With a Regional-to-Global Fusion Mechanism for Multimodal Emotion Recognition. IEEE Transactions on Affective Computing, 17(2), 1774-1787. https://doi.org/10.1109/taffc.2026.3672521

Vancouver

Zhang Y, Guo C, Zhu K, Hu W, Hu B, Shen J. R2G$^{3}$ Net: A Novel Hierarchical Spatial-Temporal Neural Network With a Regional-to-Global Fusion Mechanism for Multimodal Emotion Recognition. IEEE Transactions on Affective Computing. 2026;17(2):1774-1787. doi:10.1109/taffc.2026.3672521

BibTeX
@article{Zhang2026,
  author = {Zhang, Yanan and Guo, Chenxu and Zhu, Kexin and Hu, Wenbo and Hu, Bin and Shen, Jian},
  title = {R2G$^3$ Net: A Novel Hierarchical Spatial-Temporal Neural Network With a Regional-to-Global Fusion Mechanism for Multimodal Emotion Recognition},
  journal = {IEEE Transactions on Affective Computing},
  year = {2026},
  volume = {17},
  number = {2},
  pages = {1774--1787},
  doi = {10.1109/taffc.2026.3672521}
}
Kostenlos starten →