EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Cognition-driven Adaptive Semantic Decoding Framework for Multimodal Sentiment Analysis

Jiajia Tang, Gang Xie, Honggang Liu, Xuanyu Jin, Jianhai Zhang, Wanzeng Kong

ACM Transactions on Multimedia Computing, Communications, and Applications · 30.06.2026 · 22 (6) · 1-23

DOI: 10.1145/3802587 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 21.05.2026. Verfasst von Jiajia Tang und 5 weiteren Autor:innen. Die Arbeit nennt 60 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Zhejiang Provincial Natural Science Foundation of China, Key Research and Development Project of Zhejiang Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In real-world scenarios, multimodal sentiment analysis faces significant challenges, particularly in cross-scenario generalization. Existing works fail to effectively deal with the variability in evaluation frameworks and modality combinations, which results in poor transfer performance across different application contexts. In this article, the cognition-driven adaptive semantic decoding framework (CASDF) is proposed to realize an evaluation system and modality-independent multimodal sentiment analysis. Specifically, the adaptive modality association module is proposed to construct the adaptive modality mapping space, which allows us to dynamically adapt to arbitrary modality combinations. This indeed breaks through the limitation of the modality number and effectively deals with the modality gap. Furthermore, similar to the human hierarchical cognition (“perception-concept-decision”), the evaluation system progressive alignment module is presented to establish the unified evaluation system. This consists of the perception, concept, and decision analysis, which contributes to the adaptive cross-task analysis from the discrete sentiment space to the continuous sentiment space. The above joint analysis of the evaluation system and modality number indeed leads to the more flexible and generable multimodal sentiment semantic decoding paradigm. The experiments demonstrate that our sentiment semantic analysis network can achieve state-of-the-art performance.

Förderung

  • National Natural Science Foundation of China
  • Zhejiang Provincial Natural Science Foundation of China
  • Key Research and Development Project of Zhejiang Province

Zitieren

APA

Tang, J., Xie, G., Liu, H., Jin, X., Zhang, J., & Kong, W. (2026). Cognition-driven Adaptive Semantic Decoding Framework for Multimodal Sentiment Analysis. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(6), 1-23. https://doi.org/10.1145/3802587

Vancouver

Tang J, Xie G, Liu H, Jin X, Zhang J, Kong W. Cognition-driven Adaptive Semantic Decoding Framework for Multimodal Sentiment Analysis. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026;22(6):1-23. doi:10.1145/3802587

BibTeX
@article{Tang2026,
  author = {Tang, Jiajia and Xie, Gang and Liu, Honggang and Jin, Xuanyu and Zhang, Jianhai and Kong, Wanzeng},
  title = {Cognition-driven Adaptive Semantic Decoding Framework for Multimodal Sentiment Analysis},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2026},
  volume = {22},
  number = {6},
  pages = {1--23},
  doi = {10.1145/3802587}
}
Kostenlos starten →