EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Robust multimodal emotion recognition under missing and incomplete data with cross-modal regeneration

Behzad Mahaseni, Naimul Mefraz Khan

Empathic Computing · 2026

DOI: 10.70401/ec.2026.0023 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Empathic Computing, 2026. Verfasst von Behzad Mahaseni und 1 weiteren Autor:innen. Die Arbeit nennt 75 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Aims

Multimodal emotion recognition (MER) can outperform unimodal approaches by integrating complementary information from multiple sources. However, real-world applications often involve incomplete or missing modalities, reducing the reliability of existing MER models. This study proposes a framework that remains robust under missing-modality conditions while preserving the advantages of multimodal integration.

Methods

To address this challenge, we propose a cross-modal latent regeneration and attention long short-term memory (CMLR-ALSTM) framework. The framework combines pretrained variational autoencoder encoders with residual projection networks, optimized with L2 loss, to achieve stable latent-space alignment across modalities. Regenerated and available latent representations are integrated using a cross-modal attention mechanism and processed by an LSTM to model temporal dependencies and improve multimodal fusion under incomplete data.

Results

The proposed framework was evaluated on three benchmark datasets under complete, partially missing, and completely missing modality scenarios. Experimental results demonstrate that CMLR-ALSTM achieves up to 17.22% improvement under missing-modality conditions while maintaining competitive performance under complete-modality settings. The results also show that the proposed latent regeneration strategy effectively preserves cross-modal relationships and robust latent representations.

Conclusion

The experimental results confirm the effectiveness of the proposed framework, particularly in realistic environments where data availability is inconsistent. By leveraging CMLR to reconstruct missing representations and modelling temporal dependencies through LSTM, the proposed approach provides a more robust and reliable MER framework for practical deployment. The results across different modality combinations demonstrate its ability to generalize across heterogeneous multimodal settings.

Zitieren

APA

Mahaseni, B., & Khan, N. M. (2026). Robust multimodal emotion recognition under missing and incomplete data with cross-modal regeneration. Empathic Computing. https://doi.org/10.70401/ec.2026.0023

Vancouver

Mahaseni B, Khan NM. Robust multimodal emotion recognition under missing and incomplete data with cross-modal regeneration. Empathic Computing. 2026. doi:10.70401/ec.2026.0023

BibTeX
@article{Mahaseni2026,
  author = {Mahaseni, Behzad and Khan, Naimul Mefraz},
  title = {Robust multimodal emotion recognition under missing and incomplete data with cross-modal regeneration},
  journal = {Empathic Computing},
  year = {2026},
  doi = {10.70401/ec.2026.0023}
}
Kostenlos starten →