Dynamic Confidence-Aware Multi-Modal Emotion Recognition
IEEE Transactions on Affective Computing · 2024-07 · 15 (3) · 1358-1370
DOI: 10.1109/taffc.2023.3340924 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, 2024-07. Verfasst von Qi Zhu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 64-mal zitiert (Crossref) und nennt 56 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Natural Science Foundation of Jiangsu Province, Shenzhen Science and Technology Program.
Abstract (Originalsprache)
Multi-modal emotion recognition has attracted increasing attention in human-computer interaction, as it extracts complementary information from physiological and behavioral features. Compared to single modal approaches, multi-modal fusion methods are more susceptible to uncertainty in emotion recognition, such as heterogeneity and inconsistent predictions across different modalities. Previous multi-modal approaches ignore systematic modeling of uncertainty in fusion and revelation of dynamic variations in emotion process. In this paper, we propose a dynamic confidence-aware fusion network for robust recognition of heterogeneous emotion features, including electroencephalogram (EEG) and facial expression. First, we develop a self-attention based multi-channel LSTM network to preliminarily align the heterogeneous emotion features. Second, we propose a confidence regression network to estimate true class probability (TCP) on each modality, which helps explore the uncertainty at modality level. Then, different modalities are weighted fused according to above two types of uncertainty. Finally, we adopt self-paced learning (SPL) mechanism to further improve the model robustness by alleviating negative effect from the hard learning samples. The experimental results on several multi-modal emotion datasets demonstrate the proposed method outperforms the state-of-the-art methods in emotion recognition performance and explicitly reveals the dynamic variation of emotion with uncertainty estimation. Our code is available at:
Förderung
- National Natural Science Foundation of China
- Natural Science Foundation of Jiangsu Province
- Shenzhen Science and Technology Program
- Key Research and Development Plan of Jiangsu Province
Zitieren
APA
Zhu, Q., Zheng, C., Zhang, Z., Shao, W., & Zhang, D. (2024). Dynamic Confidence-Aware Multi-Modal Emotion Recognition. IEEE Transactions on Affective Computing, 15(3), 1358-1370. https://doi.org/10.1109/taffc.2023.3340924
Vancouver
Zhu Q, Zheng C, Zhang Z, Shao W, Zhang D. Dynamic Confidence-Aware Multi-Modal Emotion Recognition. IEEE Transactions on Affective Computing. 2024;15(3):1358-1370. doi:10.1109/taffc.2023.3340924
BibTeX
@article{Zhu2024,
author = {Zhu, Qi and Zheng, Chuhang and Zhang, Zheng and Shao, Wei and Zhang, Daoqiang},
title = {Dynamic Confidence-Aware Multi-Modal Emotion Recognition},
journal = {IEEE Transactions on Affective Computing},
year = {2024},
volume = {15},
number = {3},
pages = {1358--1370},
doi = {10.1109/taffc.2023.3340924}
}
EUVIMED