EUVIMEDEUVIMED entdecken
Crossref

ATTSF-Net: Attention-Based Similarity Fusion Network for Audio-Visual Emotion Recognition

Jiaming Zhang, Zhijia Zhang, Zhaojie Ju

IEEE Transactions on Affective Computing · 2025-10 · 16 (4) · 3291-3303

DOI: 10.1109/taffc.2025.3591567 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, 2025-10. Verfasst von Jiaming Zhang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 51 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Emotional factors play a pivotal role in fields such as autonomous driving and intelligent emotional robotics. The accurate extraction of emotional factors is instrumental in reducing error rates within these domains. With the continuous deepening of exploration in the emotional domain, rich multimodal data has progressively supplanted unimodal data. Nevertheless, current multimodal approaches still grapple with the following challenges: 1.Partial loss of information both within individual modalities and across different modalities. 2.Incorrect extraction of modality-invariant features. To facilitate multimodal interaction and address the aforementioned issues, this paper proposes an Attention-based Similarity Fusion Network (ATTSF-Net) for audio-visual emotion recognition. The network is based on multimodal data and comprises the proposed Cross-Multimodal Block (CMB), Similarity Adjustment Block (SAB), and Audio-Visual Auxiliary Modules (AVAM). CMB employs a cross-modal attention mechanism and model-level fusion to facilitate interactions between modalities. SAB is designed to learn modality-invariant features. AVAM utilizes additional audio-visual auxiliary networks to provide supplementary emotional information, enabling the full extraction of intra-modal information. A similarity loss function based on Kullback-Leibler (KL) divergence is designed to ensure the consistency of the learned audio-visual emotional information. The proposed model achieves an accuracy of 88.67% on the RAVDESS dataset (8.67% higher than human) and an unweighted accuracy (UA) of 81.93% and a weighted accuracy (WA) of 79.77% on the IEMOCAP dataset (4.01% and 5.86% higher than transfer learning, respectively). A series of visualization experiments are conducted to demonstrate the effectiveness of the proposed model.

Zitieren

APA

Zhang, J., Zhang, Z., & Ju, Z. (2025). ATTSF-Net: Attention-Based Similarity Fusion Network for Audio-Visual Emotion Recognition. IEEE Transactions on Affective Computing, 16(4), 3291-3303. https://doi.org/10.1109/taffc.2025.3591567

Vancouver

Zhang J, Zhang Z, Ju Z. ATTSF-Net: Attention-Based Similarity Fusion Network for Audio-Visual Emotion Recognition. IEEE Transactions on Affective Computing. 2025;16(4):3291-3303. doi:10.1109/taffc.2025.3591567

BibTeX
@article{Zhang2025,
  author = {Zhang, Jiaming and Zhang, Zhijia and Ju, Zhaojie},
  title = {ATTSF-Net: Attention-Based Similarity Fusion Network for Audio-Visual Emotion Recognition},
  journal = {IEEE Transactions on Affective Computing},
  year = {2025},
  volume = {16},
  number = {4},
  pages = {3291--3303},
  doi = {10.1109/taffc.2025.3591567}
}
Kostenlos starten →