Phy-FusionNet: A Memory-Augmented Transformer for Multimodal Emotion Recognition With Periodicity and Contextual Attention
IEEE Transactions on Affective Computing · 2026-01 · 17 (1) · 235-246
DOI: 10.1109/taffc.2025.3609046 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, 2026-01. Verfasst von Tianyi Wu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 58 Literaturangaben. Gefördert von: Xi’an Jiaotong-Liverpool University.
Abstract (Originalsprache)
Accurate emotion recognition from physiological signals is critical for applications in healthcare, autonomous systems, and human-computer interaction. However, prevailing methods often fail to model long-term dependencies and overlook periodic patterns inherent in physiological data. To address these challenges, we propose Phy-FusionNet, a novel memory-augmented transformer architecture for multimodal emotion recognition. Phy-FusionNet introduces a Memory Stream Module with FIFO-queue and decay-based updates to preserve long-term contextual information. It further integrates Fourier-based positional encoding and frequency-aware attention, enabling robust detection of periodic emotional cues. An Adaptive Temporal Attention Module enhances computational efficiency and enables dynamic relevance in temporal feature extraction. For cross-modal fusion, we employ a transformer-based Multimodal Binding Learning framework that balances modality-specific and shared features. Extensive experiments on five public datasets—WESAD, CL-Drive, PPB-Emo, PhyMER, and EEG-VUI—demonstrate that Phy-FusionNet outperforms state-of-the-art models, achieving up to 16.3% improvement in accuracy and superior robustness across diverse emotional states and noisy environments. Notably, the model maintains low performance variance across emotion classes, with F1-Score differences under 2.5%, indicating stable recognition even for subtle or overlapping emotions. Our results underscore the importance of integrating memory, frequency, and adaptive attention for effective affective computing. The code will be publicly available on GitHub.
Förderung
- Xi’an Jiaotong-Liverpool University
Zitieren
APA
Wu, T., Purwanto, E., Huang, Y., & Su, Y. (2026). Phy-FusionNet: A Memory-Augmented Transformer for Multimodal Emotion Recognition With Periodicity and Contextual Attention. IEEE Transactions on Affective Computing, 17(1), 235-246. https://doi.org/10.1109/taffc.2025.3609046
Vancouver
Wu T, Purwanto E, Huang Y, Su Y. Phy-FusionNet: A Memory-Augmented Transformer for Multimodal Emotion Recognition With Periodicity and Contextual Attention. IEEE Transactions on Affective Computing. 2026;17(1):235-246. doi:10.1109/taffc.2025.3609046
BibTeX
@article{Wu2026,
author = {Wu, Tianyi and Purwanto, Erick and Huang, Yongrun and Su, Yang},
title = {Phy-FusionNet: A Memory-Augmented Transformer for Multimodal Emotion Recognition With Periodicity and Contextual Attention},
journal = {IEEE Transactions on Affective Computing},
year = {2026},
volume = {17},
number = {1},
pages = {235--246},
doi = {10.1109/taffc.2025.3609046}
}
EUVIMED