Robust multimodal emotion recognition under missing and incomplete data with cross-modal regeneration
Empathic Computing · 2026
DOI: 10.70401/ec.2026.0023 ↗Kurz gefasst
Fachartikel in der Zeitschrift Empathic Computing, 2026. Verfasst von Behzad Mahaseni und 1 weiteren Autor:innen. Die Arbeit nennt 75 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Aims
Multimodal emotion recognition (MER) can outperform unimodal approaches by integrating complementary information from multiple sources. However, real-world applications often involve incomplete or missing modalities, reducing the reliability of existing MER models. This study proposes a framework that remains robust under missing-modality conditions while preserving the advantages of multimodal integration.
Methods
To address this challenge, we propose a cross-modal latent regeneration and attention long short-term memory (CMLR-ALSTM) framework. The framework combines pretrained variational autoencoder encoders with residual projection networks, optimized with L2 loss, to achieve stable latent-space alignment across modalities. Regenerated and available latent representations are integrated using a cross-modal attention mechanism and processed by an LSTM to model temporal dependencies and improve multimodal fusion under incomplete data.
Results
The proposed framework was evaluated on three benchmark datasets under complete, partially missing, and completely missing modality scenarios. Experimental results demonstrate that CMLR-ALSTM achieves up to 17.22% improvement under missing-modality conditions while maintaining competitive performance under complete-modality settings. The results also show that the proposed latent regeneration strategy effectively preserves cross-modal relationships and robust latent representations.
Conclusion
The experimental results confirm the effectiveness of the proposed framework, particularly in realistic environments where data availability is inconsistent. By leveraging CMLR to reconstruct missing representations and modelling temporal dependencies through LSTM, the proposed approach provides a more robust and reliable MER framework for practical deployment. The results across different modality combinations demonstrate its ability to generalize across heterogeneous multimodal settings.
Zitieren
APA
Mahaseni, B., & Khan, N. M. (2026). Robust multimodal emotion recognition under missing and incomplete data with cross-modal regeneration. Empathic Computing. https://doi.org/10.70401/ec.2026.0023
Vancouver
Mahaseni B, Khan NM. Robust multimodal emotion recognition under missing and incomplete data with cross-modal regeneration. Empathic Computing. 2026. doi:10.70401/ec.2026.0023
BibTeX
@article{Mahaseni2026,
author = {Mahaseni, Behzad and Khan, Naimul Mefraz},
title = {Robust multimodal emotion recognition under missing and incomplete data with cross-modal regeneration},
journal = {Empathic Computing},
year = {2026},
doi = {10.70401/ec.2026.0023}
}
EUVIMED