EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Multimodal Emotion Recognition Method Based on Domain Generalization and Graph Neural Networks

Jinbao Xie, Yulong Wang, Tianxin Meng, Jianqiao Tai, Yueqian Zheng, Yury I. Varatnitski

Electronics · 23.02.2025 · 14 (5) · 885

DOI: 10.3390/electronics14050885 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 23.02.2025. Verfasst von Jinbao Xie und 5 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Education Department of Hainan Province, Key Project of Application Research on the National Smart Education Platform for Primary and Secondary Schools in Hainan Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In recent years, multimodal sentiment analysis has attracted increasing attention from researchers owing to the rapid development of human–computer interactions. Sentiment analysis is an important task for understanding dialogues. However, with the increase of multimodal data, the processing of individual modality features and the methods for multimodal feature fusion have become more significant for research. Existing methods that handle the features of each modality separately are not suitable for subsequent multimodal fusion and often fail to capture sufficient global and local information. Therefore, this study proposes a novel multimodal sentiment analysis method based on domain generalization and graph neural networks. The main characteristic of this method is that it considers the features of each modality as domains. It extracts domain-specific and cross-domain-invariant features, thereby facilitating cross-domain generalization. Generalized features are more suitable for multimodal fusion. Graph neural networks were employed to extract global and local information from the dialogue to capture the emotional changes of the speakers. Specifically, global representations were captured by modeling cross-modal interactions at the dialogue level, whereas local information was typically inferred from temporal information or the emotional changes of the speakers. The method proposed in this study outperformed existing models on the IEMOCAP, CMU-MOSEI, and MELD datasets by 0.97%, 1.09% (for seven-class classification), and 0.65% in terms of weighted F1 score, respectively. This clearly demonstrates that the domain-generalized features proposed in this study are better suited for subsequent multimodal fusion, and that the model developed here is more effective at capturing both global and local information.

Förderung

  • Education Department of Hainan Province
  • Key Project of Application Research on the National Smart Education Platform for Primary and Secondary Schools in Hainan Province

Zitieren

APA

Xie, J., Wang, Y., Meng, T., Tai, J., Zheng, Y., & Varatnitski, Y. I. (2025). Multimodal Emotion Recognition Method Based on Domain Generalization and Graph Neural Networks. Electronics, 14(5), 885. https://doi.org/10.3390/electronics14050885

Vancouver

Xie J, Wang Y, Meng T, Tai J, Zheng Y, Varatnitski YI. Multimodal Emotion Recognition Method Based on Domain Generalization and Graph Neural Networks. Electronics. 2025;14(5):885. doi:10.3390/electronics14050885

BibTeX
@article{Xie2025,
  author = {Xie, Jinbao and Wang, Yulong and Meng, Tianxin and Tai, Jianqiao and Zheng, Yueqian and Varatnitski, Yury I.},
  title = {Multimodal Emotion Recognition Method Based on Domain Generalization and Graph Neural Networks},
  journal = {Electronics},
  year = {2025},
  volume = {14},
  number = {5},
  pages = {885},
  doi = {10.3390/electronics14050885}
}
Kostenlos starten →