Multi-modal emotion recognition using tensor decomposition fusion and self-supervised multi-tasking
International Journal of Multimedia Information Retrieval · 2024-12 · 13 (4)
DOI: 10.1007/s13735-024-00347-3 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Multimedia Information Retrieval, online erschienen am 03.09.2024. Verfasst von Rui Wang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 84-mal zitiert (Crossref) und nennt 28 Literaturangaben. Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
With technological advancements, we can now capture rich dialogue content, tones, textual information, and visual data through tools like microphones, the internet, and cameras. However, relying solely on a single modality for emotion analysis often fails to accurately reflect the true emotional state, as this approach overlooks the dynamic correlations between different modalities. To address this, our study introduces a multimodal emotion recognition method that combines tensor decomposition fusion and self-supervised multi-task learning. This method first employs Tucker decomposition techniques to effectively reduce the model’s parameter count, lowering the risk of overfitting. Subsequently, by building a learning mechanism for both multimodal and unimodal tasks and incorporating the concept of label generation, it more accurately captures the emotional differences between modalities. We conducted extensive experiments and analyses on public datasets like CMU-MOSI and CMU-MOSEI, and the results show that our method significantly outperforms existing methods in terms of performance. The related code is open-sourced at https://github.com/ZhuJw31/MMER-TD.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Wang, R., Zhu, J., Wang, S., Wang, T., Huang, J., & Zhu, X. (2024). Multi-modal emotion recognition using tensor decomposition fusion and self-supervised multi-tasking. International Journal of Multimedia Information Retrieval, 13(4). https://doi.org/10.1007/s13735-024-00347-3
Vancouver
Wang R, Zhu J, Wang S, Wang T, Huang J, Zhu X. Multi-modal emotion recognition using tensor decomposition fusion and self-supervised multi-tasking. International Journal of Multimedia Information Retrieval. 2024;13(4). doi:10.1007/s13735-024-00347-3
BibTeX
@article{Wang2024,
author = {Wang, Rui and Zhu, Jiawei and Wang, Shoujin and Wang, Tao and Huang, Jingze and Zhu, Xianxun},
title = {Multi-modal emotion recognition using tensor decomposition fusion and self-supervised multi-tasking},
journal = {International Journal of Multimedia Information Retrieval},
year = {2024},
volume = {13},
number = {4},
doi = {10.1007/s13735-024-00347-3}
}
EUVIMED