A Multi-Level Alignment and Cross-Modal Unified Semantic Graph Refinement Network for Conversational Emotion Recognition
IEEE Transactions on Affective Computing · 2024-07 · 15 (3) · 1553-1566
DOI: 10.1109/taffc.2024.3354382 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, 2024-07. Verfasst von Xiaoheng Zhang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 28-mal zitiert (Crossref) und nennt 50 Literaturangaben. Gefördert von: National Natural Science Funds for Distinguished Young Scholar, Beijing Natural Science Foundation, National Key Research and Development Program of China.
Abstract (Originalsprache)
Emotion recognition in conversation (ERC) based on multiple modalities has attracted enormous attention. However, most research simply concatenated multimodal representations, generally neglecting the impact of cross-modal correspondences and uncertain factors, and leading to the cross-modal misalignment problems. Furthermore, recent methods only considered simple contextual features, commonly ignoring semantic clues and resulting in an insufficient capture of the semantic consistency. To address these limitations, we propose a novel multi-level alignment and cross-modal unified semantic graph refinement network (MA-CMU-SGRNet) for ERC task. Specifically, a multi-level alignment (MA) is first designed to bridge the gap between acoustic and lexical modalities, which can effectively contrast both the instance-level and prototype-level relationships, separating the multimodal features in the latent space. Second, a cross-modal uncertainty-aware unification (CMU) is adopted to generate a unified representation in joint space considering the ambiguity of emotion. Finally, a dual-encoding semantic graph refinement network (SGRNet) is investigated, which includes a syntactic encoder to aggregate information from near neighbors and a semantic encoder to focus on useful semantically close neighbors. Extensive experiments on three multimodal public datasets show the effectiveness of our proposed method compared with the state-of-the-art methods, indicating its potential application in conversational emotion recognition. Implementation codes can be available athttps://github.com/zxiaohen/MA-CMU-SGRNet.
Förderung
- National Natural Science Funds for Distinguished Young Scholar
- Beijing Natural Science Foundation
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- China Postdoctoral Science Foundation
- Beijing United Imaging Research Institute of Intelligent Imaging Foundation
Zitieren
APA
Zhang, X., Cui, W., Hu, B., & Li, Y. (2024). A Multi-Level Alignment and Cross-Modal Unified Semantic Graph Refinement Network for Conversational Emotion Recognition. IEEE Transactions on Affective Computing, 15(3), 1553-1566. https://doi.org/10.1109/taffc.2024.3354382
Vancouver
Zhang X, Cui W, Hu B, Li Y. A Multi-Level Alignment and Cross-Modal Unified Semantic Graph Refinement Network for Conversational Emotion Recognition. IEEE Transactions on Affective Computing. 2024;15(3):1553-1566. doi:10.1109/taffc.2024.3354382
BibTeX
@article{Zhang2024,
author = {Zhang, Xiaoheng and Cui, Weigang and Hu, Bin and Li, Yang},
title = {A Multi-Level Alignment and Cross-Modal Unified Semantic Graph Refinement Network for Conversational Emotion Recognition},
journal = {IEEE Transactions on Affective Computing},
year = {2024},
volume = {15},
number = {3},
pages = {1553--1566},
doi = {10.1109/taffc.2024.3354382}
}
EUVIMED