EUVIMEDEUVIMED entdecken
Open Access · CC BY-SA 4.0

CNN-LSTM with Transformer-Based Fusion (CLIP: Contrastive Language-Image Pretraining) for Sentiment and Emotion

Irsyad Khalid Ilyas, Andi Sunyoto, M. Hanafi

JOINTECS (Journal of Information Technology and Computer Science) · 2025

DOI: 10.31328/jointecs.v10i1.7346 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JOINTECS (Journal of Information Technology and Computer Science), 2025. Verfasst von Irsyad Khalid Ilyas und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-SA 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Sentiment analysis on multimodal data presents significant challenges due to the complex integration of textual and visualinformation, especially in internet memes where meaning often emerges from the interplay between image and text. This studyproposes a hybrid deep learning model that combines Convolutional Neural Networks (CNN) for visual feature extraction,Long Short-Term Memory (LSTM) networks for contextual text understanding, and Contrastive Language–Image Pretraining(CLIP) as a transformer-based fusion mechanism to align both modalities in a shared semantic space. Evaluated on theMemotion 7K dataset, the proposed model achieves an accuracy of 85.6%, outperforming baseline architectures such as CNNLSTM (78.3%) and BERT-DenseNet (81.2%). Experimental results demonstrate that CLIP’s contrastive learning significantlyenhances the model’s ability to interpret ambiguous and sarcastic content by capturing nuanced text-image relationships. Thesystem shows balanced performance across sentiment classes, with strong precision and recall metrics, confirming itsrobustness. This research contributes a methodologically sound framework for multimodal sentiment analysis and offerspractical implications for real-world applications such as social media monitoring, digital marketing, and cross-culturalcontent understanding. Future work includes multilingual dataset expansion, computational optimization for deployment, andintegration of additional modalities like audio or video to further enrich emotional context.

Zitieren

APA

Ilyas, I. K., Sunyoto, A., & Hanafi, M. (2025). CNN-LSTM with Transformer-Based Fusion (CLIP: Contrastive Language-Image Pretraining) for Sentiment and Emotion. JOINTECS (Journal of Information Technology and Computer Science). https://doi.org/10.31328/jointecs.v10i1.7346

Vancouver

Ilyas IK, Sunyoto A, Hanafi M. CNN-LSTM with Transformer-Based Fusion (CLIP: Contrastive Language-Image Pretraining) for Sentiment and Emotion. JOINTECS (Journal of Information Technology and Computer Science). 2025. doi:10.31328/jointecs.v10i1.7346

BibTeX
@article{Ilyas2025,
  author = {Ilyas, Irsyad Khalid and Sunyoto, Andi and Hanafi, M.},
  title = {CNN-LSTM with Transformer-Based Fusion (CLIP: Contrastive Language-Image Pretraining) for Sentiment and Emotion},
  journal = {JOINTECS (Journal of Information Technology and Computer Science)},
  year = {2025},
  doi = {10.31328/jointecs.v10i1.7346}
}
Kostenlos starten →