EUVIMEDEUVIMED entdecken
Crossref

Domain-Separated Bottleneck Attention Fusion Framework for Multimodal Emotion Recognition

Peng He, Jun Yu, Chengjie Ge, Ye Yu, Wei Xu, Lei Wang, Tianyu Liu, Zhen Kan

ACM Transactions on Multimedia Computing, Communications, and Applications · 30.04.2025 · 21 (4) · 1-21

DOI: 10.1145/3711865 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 10.03.2025. Verfasst von Peng He und 7 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 80 Literaturangaben. Gefördert von: Natural Science Foundation of China, National Aviation Science Foundation, CAAI-Huawei MindSpore Open Fund.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

As a focal point of research in various fields, human body language understanding has long been a subject of intense interest. Within this realm, the exploration of emotion recognition through the analysis of facial expressions, voice patterns, and physiological signals holds significant practical value. Compared with unimodal approaches, multimodal emotion recognition models leverage complementary information from vision, acoustic, and language modalities to robust perceive the human sentiment attitudes. However, the heterogeneity among modality signals leads to significant domain shifts, posing challenges for achieving balanced fusion. In this article, we propose a Domain-Separated Bottleneck Attention (DBA) Fusion Framework for human multimodal emotion recognition with lower computational complexity. Specifically, we partition each modality into two distinct domains: the invariant/private domain. The invariant domain contains crucial shared information, while the private domain aims to capture modality-specific representations. For the decomposed features, we introduce two sets of bottleneck cross-attention modules to effectively utilize the complementarity between domains to reduce redundant information. In each module, we interweave two Fusion Adapter blocks into the Self-Attention Transformer backbone. Each Fusion Adapter block integrates a small group of latent tokens as bridges for inter-modal and inter-domain interactions, mitigating the adverse effects of modality distribution differences and lowering computational costs. Extensive experimental results demonstrate that our method outperforms State-of-the-Art (SOTA) approaches across three widely used benchmark datasets.

Förderung

  • Natural Science Foundation of China
  • National Aviation Science Foundation
  • CAAI-Huawei MindSpore Open Fund
  • Anhui Province Key Research and Development Program
  • Dreams Foundation of Jianghuai Advance Technology Center
  • Beijing Municipal Science and Technology Commission, and Administrative Commission of Zhongguancun Science Park

Zitieren

APA

He, P., Yu, J., Ge, C., Yu, Y., Xu, W., Wang, L., Liu, T., & Kan, Z. (2025). Domain-Separated Bottleneck Attention Fusion Framework for Multimodal Emotion Recognition. ACM Transactions on Multimedia Computing, Communications, and Applications, 21(4), 1-21. https://doi.org/10.1145/3711865

Vancouver

He P, Yu J, Ge C, Yu Y, Xu W, Wang L, et al. Domain-Separated Bottleneck Attention Fusion Framework for Multimodal Emotion Recognition. ACM Transactions on Multimedia Computing, Communications, and Applications. 2025;21(4):1-21. doi:10.1145/3711865

BibTeX
@article{He2025,
  author = {He, Peng and Yu, Jun and Ge, Chengjie and Yu, Ye and Xu, Wei and Wang, Lei and Liu, Tianyu and Kan, Zhen},
  title = {Domain-Separated Bottleneck Attention Fusion Framework for Multimodal Emotion Recognition},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2025},
  volume = {21},
  number = {4},
  pages = {1--21},
  doi = {10.1145/3711865}
}
Kostenlos starten →