Connecting Cross-Modal Representations for Compact and Robust Multimodal Sentiment Analysis With Sentiment Word Substitution Error
IEEE Transactions on Affective Computing · 2025-07 · 16 (3) · 1265-1276
DOI: 10.1109/taffc.2024.3490694 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, 2025-07. Verfasst von Qiyuan Sun und 3 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 48 Literaturangaben. Gefördert von: Young Scientists Fund, General Program, National Natural Science Foundation of China.
Abstract (Originalsprache)
Multimodal Sentiment Analysis (MSA) seeks to fuse textual, acoustic, and visual information to predict a speaker’s sentiment states effectively. However, in real-world scenarios, the text modality received by MSA systems is often obtained through automatic speech recognition (ASR) models. Unfortunately, ASR may erroneously recognize sentiment words as phonetically similar neutral alternatives, leading to sentiment degradation in text and impacting MSA accuracy. Recent attempts aim to first identify the sentiment word substitution (SWS) error in ASR results and then refine the corrupted word embeddings using multimodal information for final multimodal fusion. However, such a method includes a burdensome and ambiguous detection operation and ignores the inherent correlations and heterogeneity among different modalities. To address these issues, we propose a more compact system, termed ARF-MSA consisting of three key components to achieving robust MSA with SWS errors: 1) Alignment: we establish connections between the “text-acoustic’ and “text-visual” representations to effectively map the “text-acoustic-visual” data into a unified sentiment space by leveraging their multimodal correlation knowledge; 2) Refinement: we perform fine-grained comparisons between the text modality and the other two modalities in the unified sentiment space, enabling refinement of the sentiment expression within the text modality more concisely; 3) Fusion: Finally, we hierarchically fuse the dominant and non-dominant representation from three heterogeneity modalities to obtain the multimodal feature for MSA. We conduct extensive experiments on the real-world datasets and the results demonstrate the effectiveness of our model.
Förderung
- Young Scientists Fund
- General Program
- National Natural Science Foundation of China
- Guangdong Provincial Key Laboratory of Human Digital Twin
- Inner Mongolia Science and Technology Achievement Transfer and Transformation Demonstration Zone, University Collaborative Innovation Base, and University Entr…
- Shenzhen Science and Technology Program
- Shenzhen Science and Technology Research Fund
Zitieren
APA
Sun, Q., Zuo, H., Liu, R., & Li, H. (2025). Connecting Cross-Modal Representations for Compact and Robust Multimodal Sentiment Analysis With Sentiment Word Substitution Error. IEEE Transactions on Affective Computing, 16(3), 1265-1276. https://doi.org/10.1109/taffc.2024.3490694
Vancouver
Sun Q, Zuo H, Liu R, Li H. Connecting Cross-Modal Representations for Compact and Robust Multimodal Sentiment Analysis With Sentiment Word Substitution Error. IEEE Transactions on Affective Computing. 2025;16(3):1265-1276. doi:10.1109/taffc.2024.3490694
BibTeX
@article{Sun2025,
author = {Sun, Qiyuan and Zuo, Haolin and Liu, Rui and Li, Haizhou},
title = {Connecting Cross-Modal Representations for Compact and Robust Multimodal Sentiment Analysis With Sentiment Word Substitution Error},
journal = {IEEE Transactions on Affective Computing},
year = {2025},
volume = {16},
number = {3},
pages = {1265--1276},
doi = {10.1109/taffc.2024.3490694}
}
EUVIMED