Generalizing to Unseen Speakers: Multimodal Emotion Recognition in Conversations With Speaker Generalization
IEEE Transactions on Affective Computing · 2025-10 · 16 (4) · 3043-3054
DOI: 10.1109/taffc.2025.3566059 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, 2025-10. Verfasst von Geng Tu und 6 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 64 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Natural Science Foundation of Guangdong Province, Shenzhen Foundational Research Funding.
Abstract (Originalsprache)
Multimodal Emotion Recognition in Conversations (MERC) aims to identify the emotion expressed in each utterance within conversational videos. Current efforts are directed toward modeling speaker-sensitive context dependencies and multimodal fusion. However, they still struggle to handle utterances from unseen speakers, hampering the model's generalizability. To tackle this challenge, we propose a Speaker Generalization Framework for MERC. Specifically, we build a prototype graph to learn Speaker-based Utterance Representations (SUR), leveraging prototypes as the bridge between seen and unseen speakers. Speaker-aware Contrastive Learning (CL) is then applied to refine SUR, pulling utterances (or prototypes) from the same speaker together while pushing those from different speakers apart. Further, we introduce a prototypical graph CL to generalize SUR to unseen speakers, ensuring that the same speakers exhibit similar graph structures, while dissimilar ones differ. To further enhance model generalization, we introduce Uncertainty-based Generalization for Speakers, randomly sampling SUR statistics from the estimated Gaussian distribution and probabilistically replacing the original SUR. Experimental findings on two datasets highlight that our framework substantially improves the generalization of various MERC models, surpassing state-of-the-art methods.
Förderung
- National Natural Science Foundation of China
- Natural Science Foundation of Guangdong Province
- Shenzhen Foundational Research Funding
- PCL
- Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies
Zitieren
APA
Tu, G., Jing, R., Liang, B., Yu, Y., Yang, M., Qin, B., & Xu, R. (2025). Generalizing to Unseen Speakers: Multimodal Emotion Recognition in Conversations With Speaker Generalization. IEEE Transactions on Affective Computing, 16(4), 3043-3054. https://doi.org/10.1109/taffc.2025.3566059
Vancouver
Tu G, Jing R, Liang B, Yu Y, Yang M, Qin B, et al. Generalizing to Unseen Speakers: Multimodal Emotion Recognition in Conversations With Speaker Generalization. IEEE Transactions on Affective Computing. 2025;16(4):3043-3054. doi:10.1109/taffc.2025.3566059
BibTeX
@article{Tu2025,
author = {Tu, Geng and Jing, Ran and Liang, Bin and Yu, Yue and Yang, Min and Qin, Bing and Xu, Ruifeng},
title = {Generalizing to Unseen Speakers: Multimodal Emotion Recognition in Conversations With Speaker Generalization},
journal = {IEEE Transactions on Affective Computing},
year = {2025},
volume = {16},
number = {4},
pages = {3043--3054},
doi = {10.1109/taffc.2025.3566059}
}
EUVIMED