PAD-Guided Multimodal Hybrid Contrastive Emotion Recognition upon STEM-E2VA Dataset
Multimodal Technologies and Interaction · 02.04.2026 · 10 (4) · 38
DOI: 10.3390/mti10040038 ↗Kurz gefasst
Fachartikel in der Zeitschrift Multimodal Technologies and Interaction, online erschienen am 02.04.2026. Verfasst von Shufei Duan und 6 weiteren Autor:innen. Die Arbeit nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Youth Fund of the National Natural Science Foundation of China, Shanxi Scholarship Council of China, Natural Science Foundation of Shanxi Province, China.
Abstract (Originalsprache)
There are still challenges in speech emotion recognition, as the representation capability of single-modal information is limited, there are difficulties in capturing continuous emotional transitions in discrete emotion annotations, and the issues of modal structural differences and cross-sample alignment in multimodal fusion methods persist. To address these, this study undertakes work from both data and model perspectives. For data, a Chinese multimodal database STEM-E2VA was constructed, synchronously collecting four modalities of data: articulatory kinematics, acoustics, glottal signals, and videos. This covers seven discrete emotion categories and employs PAD continuous annotation. By integrating discrete and continuous dimensional annotations, it better represents the distinction between strong and weak emotions under the same discrete emotion label. Concurrently, to process the biases in PAD annotations, we employed the SCL-90 psychological questionnaire to analyze annotators’ cognitive and emotional perceptions, thereby ensuring data reliability. For model, this paper proposes a multimodal supervised contrastive fusion network incorporating PAD perception. It employs a PAD-enhanced hybrid contrastive loss function to optimize intra-model and inter-modal feature alignment. Utilizing a cross-attention mechanism combined with a GRU–Transformer network for temporal feature extraction, it achieves deep fusion of multimodal information, reducing inter-modal discrepancies and cross-class confusion. Experiments demonstrate that the proposed method achieves 85.47% accuracy in discrete sentiment recognition on STEM-E2VA, with a substantial reduction in RMSE for PAD dimension prediction. It also exhibits excellent generalization capability on IEMOCAP, providing a novel framework for integrating discrete and continuous sentiment representations.
Förderung
- Youth Fund of the National Natural Science Foundation of China
- Shanxi Scholarship Council of China
- Natural Science Foundation of Shanxi Province, China
- Startup Fund of Shanxi University of Electronic Science and Technology
Zitieren
APA
Duan, S., Zhang, W., Li, L., Zhu, T., Zhao, F., Li, F., & Liang, H. (2026). PAD-Guided Multimodal Hybrid Contrastive Emotion Recognition upon STEM-E2VA Dataset. Multimodal Technologies and Interaction, 10(4), 38. https://doi.org/10.3390/mti10040038
Vancouver
Duan S, Zhang W, Li L, Zhu T, Zhao F, Li F, et al. PAD-Guided Multimodal Hybrid Contrastive Emotion Recognition upon STEM-E2VA Dataset. Multimodal Technologies and Interaction. 2026;10(4):38. doi:10.3390/mti10040038
BibTeX
@article{Duan2026,
author = {Duan, Shufei and Zhang, Wenjie and Li, Liangqi and Zhu, Ting and Zhao, Fangyu and Li, Fujiang and Liang, Huizhi},
title = {PAD-Guided Multimodal Hybrid Contrastive Emotion Recognition upon STEM-E2VA Dataset},
journal = {Multimodal Technologies and Interaction},
year = {2026},
volume = {10},
number = {4},
pages = {38},
doi = {10.3390/mti10040038}
}
EUVIMED