Multimodal emotion recognition with disentangled representations: private-shared multimodal variational autoencoder and long short-term memory framework
Empathic Computing · 29.06.2025
DOI: 10.70401/ec.2025.0010 ↗Kurz gefasst
Fachartikel in der Zeitschrift Empathic Computing, online erschienen am 29.06.2025. Verfasst von Behzad Mahaseni und 1 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 76 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Aims
This study proposes a multimodal emotion recognition framework that combines a private-shared disentangled multimodal variational autoencoder (DMMVAE) with a long short-term memory (LSTM) network, herein referred to as DMMVAE-LSTM. The primary objective is to improve the robustness and generalizability of emotion recognition by effectively leveraging the complementary features of electroencephalogram (EEG) signals and facial expression data.
Methods
We first trained a variational autoencoder using a ResNet-101 architecture on a large-scale facial dataset to develop a robust and generalizable facial feature extractor. This pre-trained model was then integrated into the DMMVAE framework, together with a convolutional neural network-based encoder and decoder for EEG data. The DMMVAE model was trained to disentangle shared and modality-specific latent representations across both EEG and facial data. Following this, the outputs of the encoders were concatenated and fed into a LSTM classifier for emotion recognition.
Results
Two sets of experiments were conducted. First, we trained and evaluated our model on the full dataset, comparing its performance with state-of-the-art methods and a baseline LSTM model employing a late fusion strategy to combine EEG and facial features. Second, to assess robustness, we tested the DMMVAE-LSTM framework under data-limited and modality dropout conditions by training with partial data and simulating missing modalities. The results demonstrate that the DMMVAE-LSTM framework consistently outperforms the baseline, especially in scenarios with limited data, indicating its capacity to learn structured and resilient latent representations.
Conclusion
Our findings underscore the benefits of multimodal generative modeling for emotion recognition, particularly in enhancing classification performance when training data are scarce or partially missing. By effectively learning both shared and private representations, DMMVAE-LSTM framework facilitates more reliable emotion classification and presents a promising solution for real-world applications where acquiring large labeled datasets is challenging.
Zitieren
APA
Mahaseni, B., & Khan, N. M. (2025). Multimodal emotion recognition with disentangled representations: private-shared multimodal variational autoencoder and long short-term memory framework. Empathic Computing. https://doi.org/10.70401/ec.2025.0010
Vancouver
Mahaseni B, Khan NM. Multimodal emotion recognition with disentangled representations: private-shared multimodal variational autoencoder and long short-term memory framework. Empathic Computing. 2025. doi:10.70401/ec.2025.0010
BibTeX
@article{Mahaseni2025,
author = {Mahaseni, Behzad and Khan, Naimul Mefraz},
title = {Multimodal emotion recognition with disentangled representations: private-shared multimodal variational autoencoder and long short-term memory framework},
journal = {Empathic Computing},
year = {2025},
doi = {10.70401/ec.2025.0010}
}
EUVIMED