Advancing multimodal emotion analysis: a hybrid deep learning approach with intermediate fusion and multi-task learning
The Visual Computer · 2026-04 · 42 (6)
DOI: 10.1007/s00371-026-04475-1 ↗Kurz gefasst
Fachartikel in der Zeitschrift The Visual Computer, online erschienen am 14.04.2026. Verfasst von Fatih Kaya und 3 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 56 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Malatya Turgut Özal University.
Abstract (Originalsprache)
Emotion analysis is a critical research domain focused on detecting the emotional states of individuals or communities across multiple data modalities, including text, images, and audio. While substantial progress has been made in unimodal (text-based) sentiment analysis, real-world scenarios often involve multimodal data, making integrated approaches essential for capturing contextual richness and improving predictive accuracy. This study introduces a hybrid deep learning model that combines text and visual features through an intermediate fusion mechanism and multi-task learning framework. Textual inputs are processed using RoBERTa and BiGRU layers, while visual inputs are analyzed through ViT and ResNet50 architectures enhanced by the Convolutional Block Attention Module (CBAM). The fused multimodal representations enable simultaneous and more robust emotion classification. Experimental results on the MVSA dataset demonstrate the superior performance of the proposed model, achieving 96.02% accuracy, 95.51% precision, 94.07% recall, and 94.73% F1-score, outperforming several state-of-the-art multimodal benchmarks. These findings underscore the model’s methodological contributions and its strong potential for advancing the field of multimodal emotion analysis in both academic research and real-world applications.
Förderung
- Malatya Turgut Özal University
Zitieren
APA
Kaya, F., Karaca, Y. E., Aslan, S., & Yıldırım, M. (2026). Advancing multimodal emotion analysis: a hybrid deep learning approach with intermediate fusion and multi-task learning. The Visual Computer, 42(6). https://doi.org/10.1007/s00371-026-04475-1
Vancouver
Kaya F, Karaca YE, Aslan S, Yıldırım M. Advancing multimodal emotion analysis: a hybrid deep learning approach with intermediate fusion and multi-task learning. The Visual Computer. 2026;42(6). doi:10.1007/s00371-026-04475-1
BibTeX
@article{Kaya2026,
author = {Kaya, Fatih and Karaca, Yunus Emre and Aslan, Serpil and Yıldırım, Muhammed},
title = {Advancing multimodal emotion analysis: a hybrid deep learning approach with intermediate fusion and multi-task learning},
journal = {The Visual Computer},
year = {2026},
volume = {42},
number = {6},
doi = {10.1007/s00371-026-04475-1}
}
EUVIMED