EUVIMEDOdkrijte EUVIMED
Crossref

Multimodal Sentiment Analysis Based on Attentional Temporal Convolutional Network and Multi-Layer Feature Fusion

Hongju Cheng, Zizhen Yang, Xiaoqi Zhang, Yang Yang

IEEE Transactions on Affective Computing · 2023-10-01 · 14 (4) · 3149-3163

DOI: 10.1109/taffc.2023.3265653 ↗

At a glance

Journal article in IEEE Transactions on Affective Computing, published on 2023-10-01. Written by Hongju Cheng and 3 co-authors. Cited 64 times so far (Crossref); 48 references. Funded by: National Science Foundation of China.

AbstraktBibliografické údajeCiteRelated studies

Abstract (original language)

Multimodal sentiment analysis aims to extract and integrate information from different modalities to accurately identify the sentiment expressed in multimodal data. How to effectively capture the relevant information within a specific modality and how to fully exploit the complementary information among multiple modalities are two major challenges in multimodal sentiment analysis. Traditional approaches fail to obtain the global contextual information of long time-series data when extracting unimodal temporal features, and they usually fuse the features from multiple modalities with the same method and ignore the correlation between different modalities when modeling inter-modal interactions. In this paper, we first propose an Attentional Temporal Convolutional Network (ATCN) to extract unimodal temporal features for enhancing the feature representation ability, then introduce a Multi-layer Feature Fusion (MFF) model to improve the effectiveness of multimodal fusion, which fuses the different-level features by different methods according to the correlation coefficient between the features, and cross-modal multi-head attention is used to fully explore the potential relationship between the low-level features. The experimental results on SIMS and CMU-MOSI datasets show that the proposed model achieves superior performance on sentiment analysis tasks compared to state-of-the-art baselines.

Funding

  • National Science Foundation of China

Cite

APA

Cheng, H., Yang, Z., Zhang, X., & Yang, Y. (2023). Multimodal Sentiment Analysis Based on Attentional Temporal Convolutional Network and Multi-Layer Feature Fusion. IEEE Transactions on Affective Computing, 14(4), 3149-3163. https://doi.org/10.1109/taffc.2023.3265653

Vancouver

Cheng H, Yang Z, Zhang X, Yang Y. Multimodal Sentiment Analysis Based on Attentional Temporal Convolutional Network and Multi-Layer Feature Fusion. IEEE Transactions on Affective Computing. 2023;14(4):3149-3163. doi:10.1109/taffc.2023.3265653

BibTeX
@article{Cheng2023,
  author = {Cheng, Hongju and Yang, Zizhen and Zhang, Xiaoqi and Yang, Yang},
  title = {Multimodal Sentiment Analysis Based on Attentional Temporal Convolutional Network and Multi-Layer Feature Fusion},
  journal = {IEEE Transactions on Affective Computing},
  year = {2023},
  volume = {14},
  number = {4},
  pages = {3149--3163},
  doi = {10.1109/taffc.2023.3265653}
}
Start for free →