Multi-Modal Sentiment Analysis Based on Image and Text Fusion Based on Cross-Attention Mechanism
Electronics · 27.05.2024 · 13 (11) · 2069
DOI: 10.3390/electronics13112069 ↗Kurz gefasst
Fachartikel in der Zeitschrift Electronics, online erschienen am 27.05.2024. Verfasst von Hongchan Li und 2 weiteren Autor:innen. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Henan Provincial Science and Technology Project.
Abstract (Originalsprache)
Research on uni-modal sentiment analysis has achieved great success, but emotions in real life are mostly multi-modal; there are not only texts but also images, audio, video, and other forms. The various modes play a role in mutual promotion. If the connection between various modalities can be mined, the accuracy of sentiment analysis will be further improved. To this end, this paper introduces a cross-attention-based multi-modal fusion model for images and text, namely, MCAM. First, we use the ALBert pre-training model to extract text features for text; then, we use BiLSTM to extract text context features; then, we use DenseNet121 to extract image features for images; and then, we use CBAM to extract specific areas related to emotion in images. Finally, we utilize multi-modal cross-attention to fuse the extracted features from the text and image, and we classify the output to determine the emotional polarity. In the experimental comparative analysis of MVSA and TumEmo public datasets, the model in this article is better than the baseline model, with accuracy and F1 scores reaching 86.5% and 75.3% and 85.5% and 76.7%, respectively. In addition, we also conducted ablation experiments, which confirmed that sentiment analysis with multi-modal fusion is better than single-modal sentiment analysis.
Förderung
- Henan Provincial Science and Technology Project
Zitieren
APA
Li, H., Lu, Y., & Zhu, H. (2024). Multi-Modal Sentiment Analysis Based on Image and Text Fusion Based on Cross-Attention Mechanism. Electronics, 13(11), 2069. https://doi.org/10.3390/electronics13112069
Vancouver
Li H, Lu Y, Zhu H. Multi-Modal Sentiment Analysis Based on Image and Text Fusion Based on Cross-Attention Mechanism. Electronics. 2024;13(11):2069. doi:10.3390/electronics13112069
BibTeX
@article{Li2024,
author = {Li, Hongchan and Lu, Yantong and Zhu, Haodong},
title = {Multi-Modal Sentiment Analysis Based on Image and Text Fusion Based on Cross-Attention Mechanism},
journal = {Electronics},
year = {2024},
volume = {13},
number = {11},
pages = {2069},
doi = {10.3390/electronics13112069}
}
EUVIMED