EUVIMEDEUVIMED entdecken
Crossref

A Low-Rank Matching Attention Based Cross-Modal Feature Fusion Method for Conversational Emotion Recognition

Yuntao Shou, Huan Liu, Xiangyong Cao, Deyu Meng, Bo Dong

IEEE Transactions on Affective Computing · 2025-04 · 16 (2) · 1177-1189

DOI: 10.1109/taffc.2024.3498443 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, 2025-04. Verfasst von Yuntao Shou und 4 weiteren Autor:innen. Die Arbeit wurde bisher 39-mal zitiert (Crossref) und nennt 48 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Natural Science Foundation of Shaanxi Province, Project of China Knowledge Center for Engineering Science and Technology.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Conversational emotion recognition (CER) is an important research topic in human-computer interactions. Although recent advancements in transformer-based cross-modal fusion methods have shown promise in CER tasks, they tend to overlook the crucial intra-modal and inter-modal emotional interaction or suffer from high computational complexity. To address this, we introduce a novel and lightweight cross-modal feature fusion method called Low-Rank Matching Attention Method (LMAM). LMAM effectively captures contextual emotional semantic information in conversations while mitigating the quadratic complexity issue caused by the self-attention mechanism. Specifically, by setting a matching weight and calculating inter-modal features attention scores row by row, LMAM requires only one-third of the parameters of self-attention methods. We also employ the low-rank decomposition method on the weights to further reduce the number of parameters in LMAM. As a result, LMAM offers a lightweight model while avoiding overfitting problems caused by a large number of parameters. Moreover, LMAM is able to fully exploit the intra-modal emotional contextual information within each modality and integrates complementary emotional semantic information across modalities by computing and fusing similarities of intra-modal and inter-modal features simultaneously. Experimental results verify the superiority of LMAM compared with other popular cross-modal fusion methods on the premise of being more lightweight. Also, LMAM can be embedded into any existing state-of-the-art CER methods in a plug-and-play manner, and can be applied to other multi-modal recognition tasks, e.g., session recommendation and humour detection, demonstrating its remarkable generalization ability.

Förderung

  • National Natural Science Foundation of China
  • Natural Science Foundation of Shaanxi Province
  • Project of China Knowledge Center for Engineering Science and Technology
  • Wong Education Foundation

Zitieren

APA

Shou, Y., Liu, H., Cao, X., Meng, D., & Dong, B. (2025). A Low-Rank Matching Attention Based Cross-Modal Feature Fusion Method for Conversational Emotion Recognition. IEEE Transactions on Affective Computing, 16(2), 1177-1189. https://doi.org/10.1109/taffc.2024.3498443

Vancouver

Shou Y, Liu H, Cao X, Meng D, Dong B. A Low-Rank Matching Attention Based Cross-Modal Feature Fusion Method for Conversational Emotion Recognition. IEEE Transactions on Affective Computing. 2025;16(2):1177-1189. doi:10.1109/taffc.2024.3498443

BibTeX
@article{Shou2025,
  author = {Shou, Yuntao and Liu, Huan and Cao, Xiangyong and Meng, Deyu and Dong, Bo},
  title = {A Low-Rank Matching Attention Based Cross-Modal Feature Fusion Method for Conversational Emotion Recognition},
  journal = {IEEE Transactions on Affective Computing},
  year = {2025},
  volume = {16},
  number = {2},
  pages = {1177--1189},
  doi = {10.1109/taffc.2024.3498443}
}
Kostenlos starten →