EUVIMEDEUVIMED entdecken
Crossref

Connecting Cross-Modal Representations for Compact and Robust Multimodal Sentiment Analysis With Sentiment Word Substitution Error

Qiyuan Sun, Haolin Zuo, Rui Liu, Haizhou Li

IEEE Transactions on Affective Computing · 2025-07 · 16 (3) · 1265-1276

DOI: 10.1109/taffc.2024.3490694 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, 2025-07. Verfasst von Qiyuan Sun und 3 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 48 Literaturangaben. Gefördert von: Young Scientists Fund, General Program, National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal Sentiment Analysis (MSA) seeks to fuse textual, acoustic, and visual information to predict a speaker’s sentiment states effectively. However, in real-world scenarios, the text modality received by MSA systems is often obtained through automatic speech recognition (ASR) models. Unfortunately, ASR may erroneously recognize sentiment words as phonetically similar neutral alternatives, leading to sentiment degradation in text and impacting MSA accuracy. Recent attempts aim to first identify the sentiment word substitution (SWS) error in ASR results and then refine the corrupted word embeddings using multimodal information for final multimodal fusion. However, such a method includes a burdensome and ambiguous detection operation and ignores the inherent correlations and heterogeneity among different modalities. To address these issues, we propose a more compact system, termed ARF-MSA consisting of three key components to achieving robust MSA with SWS errors: 1) Alignment: we establish connections between the “text-acoustic’ and “text-visual” representations to effectively map the “text-acoustic-visual” data into a unified sentiment space by leveraging their multimodal correlation knowledge; 2) Refinement: we perform fine-grained comparisons between the text modality and the other two modalities in the unified sentiment space, enabling refinement of the sentiment expression within the text modality more concisely; 3) Fusion: Finally, we hierarchically fuse the dominant and non-dominant representation from three heterogeneity modalities to obtain the multimodal feature for MSA. We conduct extensive experiments on the real-world datasets and the results demonstrate the effectiveness of our model.

Förderung

  • Young Scientists Fund
  • General Program
  • National Natural Science Foundation of China
  • Guangdong Provincial Key Laboratory of Human Digital Twin
  • Inner Mongolia Science and Technology Achievement Transfer and Transformation Demonstration Zone, University Collaborative Innovation Base, and University Entr…
  • Shenzhen Science and Technology Program
  • Shenzhen Science and Technology Research Fund

Zitieren

APA

Sun, Q., Zuo, H., Liu, R., & Li, H. (2025). Connecting Cross-Modal Representations for Compact and Robust Multimodal Sentiment Analysis With Sentiment Word Substitution Error. IEEE Transactions on Affective Computing, 16(3), 1265-1276. https://doi.org/10.1109/taffc.2024.3490694

Vancouver

Sun Q, Zuo H, Liu R, Li H. Connecting Cross-Modal Representations for Compact and Robust Multimodal Sentiment Analysis With Sentiment Word Substitution Error. IEEE Transactions on Affective Computing. 2025;16(3):1265-1276. doi:10.1109/taffc.2024.3490694

BibTeX
@article{Sun2025,
  author = {Sun, Qiyuan and Zuo, Haolin and Liu, Rui and Li, Haizhou},
  title = {Connecting Cross-Modal Representations for Compact and Robust Multimodal Sentiment Analysis With Sentiment Word Substitution Error},
  journal = {IEEE Transactions on Affective Computing},
  year = {2025},
  volume = {16},
  number = {3},
  pages = {1265--1276},
  doi = {10.1109/taffc.2024.3490694}
}
Kostenlos starten →