Bimodal Speech Emotion Recognition Using a Hybrid CNN-LSTM Architecture with Sentiment Fusion
Future Internet · 10.08.2026 · 18 (8) · 421
DOI: 10.3390/fi18080421 ↗Kurz gefasst
Fachartikel in der Zeitschrift Future Internet, online erschienen am 10.08.2026. Verfasst von Tze-Syn Yap und 1 weiteren Autor:innen. Die Arbeit nennt 18 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Speech emotion recognition (SER) is a fundamental task in affective computing; however, traditional unimodal approaches often struggle to capture the complex emotional cues present in spontaneous conversational speech. Bimodal frameworks that integrate acoustic and textual information have therefore emerged to provide complementary semantic and acoustic representations. This study proposes a bimodal SER framework based on a hybrid convolutional neural network–long short-term memory (CNN–LSTM) architecture. Using the Multimodal EmotionLines Dataset (MELD), the framework combines temporal acoustic features, statistical acoustic features, and predicted textual sentiment. Experimental results indicate that the proposed model achieves reliable recognition of majority emotion classes but exhibits limited performance on underrepresented minority classes due to severe class imbalance. To better understand the contribution of each modality, feature sufficiency and feature necessity analyses were conducted. Furthermore, an evaluation of alternative fusion strategies showed that the expressive attention networks did not provide meaningful performance improvements over simple feature concatenation. These findings suggest that class imbalance, rather than fusion complexity, remains the primary limitation in conversational SER, highlighting the importance of addressing data imbalance before pursuing more sophisticated multimodal architectures.
Zitieren
APA
Yap, T., & Ong, L. (2026). Bimodal Speech Emotion Recognition Using a Hybrid CNN-LSTM Architecture with Sentiment Fusion. Future Internet, 18(8), 421. https://doi.org/10.3390/fi18080421
Vancouver
Yap T, Ong L. Bimodal Speech Emotion Recognition Using a Hybrid CNN-LSTM Architecture with Sentiment Fusion. Future Internet. 2026;18(8):421. doi:10.3390/fi18080421
BibTeX
@article{Yap2026,
author = {Yap, Tze-Syn and Ong, Lee-Yeng},
title = {Bimodal Speech Emotion Recognition Using a Hybrid CNN-LSTM Architecture with Sentiment Fusion},
journal = {Future Internet},
year = {2026},
volume = {18},
number = {8},
pages = {421},
doi = {10.3390/fi18080421}
}
EUVIMED