Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognition
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-10 · 47 (10) · 8786-8803
DOI: 10.1109/tpami.2025.3581236 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-10. Verfasst von Jiang Li und 2 weiteren Autor:innen. Hauptthemen laut MEDLINE: Emotions, Pattern Recognition, Automated, Natural Language Processing. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 66 Literaturangaben. Gefördert von: Interdisciplinary Research Program of HUST, National Natural Science Foundation of China, Outstanding Research Groups of Hubei Province of China.
Abstract (Originalsprache)
Multimodal emotion recognition in conversation (MERC) has garnered substantial research attention recently. Existing MERC methods face several challenges: (1) they fail to fully harness direct inter-modal cues, possibly leading to less-than-thorough cross-modal modeling; (2) they concurrently extract information from the same and different modalities at each network layer, potentially triggering conflicts from the fusion of multi-source data; (3) they lack the agility required to detect dynamic sentimental changes, perhaps resulting in inaccurate classification of utterances with abrupt sentiment shifts. To address these issues, a novel approach named GraphSmile is proposed for tracking intricate emotional cues in multimodal dialogues. GraphSmile comprises two key components, i.e., GSF and SDP modules. GSF ingeniously leverages graph structures to alternately assimilate inter-modal and intra-modal emotional dependencies layer by layer, adequately capturing cross-modal cues while effectively circumventing fusion conflicts. SDP is an auxiliary task to explicitly delineate the sentiment dynamics between utterances, promoting the model's ability to distinguish sentimental discrepancies. GraphSmile is effortlessly applied to multimodal sentiment analysis in conversation (MSAC), thus enabling simultaneous execution of MERC and MSAC tasks. Empirical results on multiple benchmarks demonstrate that GraphSmile can handle complex emotional and sentimental patterns, significantly outperforming baseline models.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Förderung
- Interdisciplinary Research Program of HUST
- National Natural Science Foundation of China
- Outstanding Research Groups of Hubei Province of China
- 111 Project on Computational Intelligence and Intelligent Control
- Fundamental Research Funds for the Central Universities of HUST
Zitieren
APA
Li, J., Wang, X., & Zeng, Z. (2025). Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(10), 8786-8803. https://doi.org/10.1109/tpami.2025.3581236
Vancouver
Li J, Wang X, Zeng Z. Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(10):8786-8803. doi:10.1109/tpami.2025.3581236
BibTeX
@article{Li2025,
author = {Li, Jiang and Wang, Xiaoping and Zeng, Zhigang},
title = {Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognition},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2025},
volume = {47},
number = {10},
pages = {8786--8803},
doi = {10.1109/tpami.2025.3581236}
}
EUVIMED