Joint learning across multi-source datasets for multi-modal emotion recognition under missing modalities
Journal of Computational Design and Engineering · 28.02.2026 · 13 (3) · 84-96
DOI: 10.1093/jcde/qwag005 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Computational Design and Engineering, online erschienen am 24.01.2026. Verfasst von Zhencheng Li und 5 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Guangdong Basic and Applied Basic Research Foundation.
Abstract (Originalsprache)
Missing modalities frequently occur in EEG- and eye movement-based emotion recognition due to the inherent instability of EEG signals and the complexity of multimodal data acquisition. Such modality absence leads to inefficient data utilization and substantial performance degradation, especially when missing modalities appear simultaneously across multiple data sets. This issue is further exacerbated when a substantial portion of a modality is missing within one or more datasets. Consequently, effectively exploiting the remaining multi-modal information under cross-dataset and missing-modality settings remains a critical and unresolved challenge. To address these challenges, we propose a novel framework termed the data generation and cross-dataset learning network (DGCDLNet), which makes the first attempt to simultaneously integrate the data generation strategy and cross-dataset learning mechanism in a unified manner. DGCDLNet contains two key modules: (1) feature reconstruction and fusion module, which leverages complete eye movement signals to compensate for missing EEG data and constructs discriminative multi-modal features via a dual-stream attention mechanism; and (2) cross-dataset learning module, which jointly learns coarse-grained representations across data sets while incorporating fine-grained features from the target-task dataset to improve classification accuracy. Extensive experiments on SEED, SEED-IV, and SEED-V demonstrate that DGCDLNet consistently outperforms state-of-the-art multi-modal fusion methods and achieves satisfactory performance under various EEG missing ratios. These results indicate the potential of DGCDLNet to advance EEG-based multi-modal emotion recognition beyond controlled laboratory settings toward practical real-world applications.
Förderung
- National Natural Science Foundation of China
- Guangdong Basic and Applied Basic Research Foundation
Zitieren
APA
Li, Z., Zhong, S., Wen, Y., Li, C., Vong, C., & Chen, C. (2026). Joint learning across multi-source datasets for multi-modal emotion recognition under missing modalities. Journal of Computational Design and Engineering, 13(3), 84-96. https://doi.org/10.1093/jcde/qwag005
Vancouver
Li Z, Zhong S, Wen Y, Li C, Vong C, Chen C. Joint learning across multi-source datasets for multi-modal emotion recognition under missing modalities. Journal of Computational Design and Engineering. 2026;13(3):84-96. doi:10.1093/jcde/qwag005
BibTeX
@article{Li2026,
author = {Li, Zhencheng and Zhong, Sheng-hua and Wen, Yimin and Li, Chen and Vong, Chi-Man and Chen, Chuangquan},
title = {Joint learning across multi-source datasets for multi-modal emotion recognition under missing modalities},
journal = {Journal of Computational Design and Engineering},
year = {2026},
volume = {13},
number = {3},
pages = {84--96},
doi = {10.1093/jcde/qwag005}
}
EUVIMED