CFN-ESA: A Cross-Modal Fusion Network With Emotion-Shift Awareness for Dialogue Emotion Recognition
IEEE Transactions on Affective Computing · 2024-10 · 15 (4) · 1919-1933
DOI: 10.1109/taffc.2024.3389453 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, 2024-10. Verfasst von Jiang Li und 3 weiteren Autor:innen. Die Arbeit wurde bisher 77-mal zitiert (Crossref) und nennt 51 Literaturangaben. Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
Multimodal emotion recognition in conversation (ERC) has garnered growing attention from research communities in various fields. In this paper, we propose a Crossmodal Fusion Network with Emotion-Shift Awareness (CFNESA) for ERC. Extant approaches employ each modality equally without distinguishing the amount of emotional information in these modalities, rendering it hard to adequately extract complementary information from multimodal data. To cope with this problem, in CFN-ESA, we treat textual modality as the primary source of emotional information, while visual and acoustic modalities are taken as the secondary sources. Besides, most multimodal ERC models ignore emotion-shift information and overfocus on contextual information, leading to the failure of emotion recognition under emotion-shift scenario. We elaborate an emotion-shift module to address this challenge. CFNESA mainly consists of unimodal encoder (RUME), cross-modal encoder (ACME), and emotion-shift module (LESM). RUME is applied to extract conversation-level contextual emotional cues while pulling together data distributions between modalities; ACME is utilized to perform multimodal interaction centered on textual modality; LESM is used to model emotion shift and capture emotion-shift information, thereby guiding the learning of the main task. Experimental results demonstrate that CFN-ESA can effectively promote performance for ERC and remarkably outperform state-of-the-art models.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Li, J., Wang, X., Liu, Y., & Zeng, Z. (2024). CFN-ESA: A Cross-Modal Fusion Network With Emotion-Shift Awareness for Dialogue Emotion Recognition. IEEE Transactions on Affective Computing, 15(4), 1919-1933. https://doi.org/10.1109/taffc.2024.3389453
Vancouver
Li J, Wang X, Liu Y, Zeng Z. CFN-ESA: A Cross-Modal Fusion Network With Emotion-Shift Awareness for Dialogue Emotion Recognition. IEEE Transactions on Affective Computing. 2024;15(4):1919-1933. doi:10.1109/taffc.2024.3389453
BibTeX
@article{Li2024,
author = {Li, Jiang and Wang, Xiaoping and Liu, Yingjian and Zeng, Zhigang},
title = {CFN-ESA: A Cross-Modal Fusion Network With Emotion-Shift Awareness for Dialogue Emotion Recognition},
journal = {IEEE Transactions on Affective Computing},
year = {2024},
volume = {15},
number = {4},
pages = {1919--1933},
doi = {10.1109/taffc.2024.3389453}
}
EUVIMED