Domain-Separated Bottleneck Attention Fusion Framework for Multimodal Emotion Recognition
ACM Transactions on Multimedia Computing, Communications, and Applications · 30.04.2025 · 21 (4) · 1-21
DOI: 10.1145/3711865 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 10.03.2025. Verfasst von Peng He und 7 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 80 Literaturangaben. Gefördert von: Natural Science Foundation of China, National Aviation Science Foundation, CAAI-Huawei MindSpore Open Fund.
Abstract (Originalsprache)
As a focal point of research in various fields, human body language understanding has long been a subject of intense interest. Within this realm, the exploration of emotion recognition through the analysis of facial expressions, voice patterns, and physiological signals holds significant practical value. Compared with unimodal approaches, multimodal emotion recognition models leverage complementary information from vision, acoustic, and language modalities to robust perceive the human sentiment attitudes. However, the heterogeneity among modality signals leads to significant domain shifts, posing challenges for achieving balanced fusion. In this article, we propose a Domain-Separated Bottleneck Attention (DBA) Fusion Framework for human multimodal emotion recognition with lower computational complexity. Specifically, we partition each modality into two distinct domains: the invariant/private domain. The invariant domain contains crucial shared information, while the private domain aims to capture modality-specific representations. For the decomposed features, we introduce two sets of bottleneck cross-attention modules to effectively utilize the complementarity between domains to reduce redundant information. In each module, we interweave two Fusion Adapter blocks into the Self-Attention Transformer backbone. Each Fusion Adapter block integrates a small group of latent tokens as bridges for inter-modal and inter-domain interactions, mitigating the adverse effects of modality distribution differences and lowering computational costs. Extensive experimental results demonstrate that our method outperforms State-of-the-Art (SOTA) approaches across three widely used benchmark datasets.
Förderung
- Natural Science Foundation of China
- National Aviation Science Foundation
- CAAI-Huawei MindSpore Open Fund
- Anhui Province Key Research and Development Program
- Dreams Foundation of Jianghuai Advance Technology Center
- Beijing Municipal Science and Technology Commission, and Administrative Commission of Zhongguancun Science Park
Zitieren
APA
He, P., Yu, J., Ge, C., Yu, Y., Xu, W., Wang, L., Liu, T., & Kan, Z. (2025). Domain-Separated Bottleneck Attention Fusion Framework for Multimodal Emotion Recognition. ACM Transactions on Multimedia Computing, Communications, and Applications, 21(4), 1-21. https://doi.org/10.1145/3711865
Vancouver
He P, Yu J, Ge C, Yu Y, Xu W, Wang L, et al. Domain-Separated Bottleneck Attention Fusion Framework for Multimodal Emotion Recognition. ACM Transactions on Multimedia Computing, Communications, and Applications. 2025;21(4):1-21. doi:10.1145/3711865
BibTeX
@article{He2025,
author = {He, Peng and Yu, Jun and Ge, Chengjie and Yu, Ye and Xu, Wei and Wang, Lei and Liu, Tianyu and Kan, Zhen},
title = {Domain-Separated Bottleneck Attention Fusion Framework for Multimodal Emotion Recognition},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2025},
volume = {21},
number = {4},
pages = {1--21},
doi = {10.1145/3711865}
}
EUVIMED