EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

GBV-Net: Hierarchical Fusion of Facial Expressions and Physiological Signals for Multimodal Emotion Recognition

Jiling Yu, Yandong Ru, Bangjun Lei, Hongming Chen iD

Sensors · 2025

DOI: 10.3390/s25206397 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, 2025. Verfasst von Jiling Yu und 3 weiteren Autor:innen. Beteiligte Einrichtung: School of Information Engineering, Zhejiang Ocean University, Zhoushan 316022, China. Hauptthemen laut MEDLINE: Emotions, Facial Expression. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

A core challenge in multimodal emotion recognition lies in the precise capture of the inherent multimodal interactive nature of human emotions. Addressing the limitation of existing methods, which often process visual signals (facial expressions) and physiological signals (EEG, ECG, EOG, and GSR) in isolation and thus fail to exploit their complementary strengths effectively, this paper presents a new multimodal emotion recognition framework called the Gated Biological Visual Network (GBV-Net). This framework enhances emotion recognition accuracy through deep synergistic fusion of facial expressions and physiological signals. GBV-Net integrates three core modules: (1) a facial feature extractor based on a modified ConvNeXt V2 architecture incorporating lightweight Transformers, specifically designed to capture subtle spatio-temporal dynamics in facial expressions; (2) a hybrid physiological feature extractor combining 1D convolutions, Temporal Convolutional Networks (TCNs), and convolutional self-attention mechanisms, adept at modeling local patterns and long-range temporal dependencies in physiological signals; and (3) an enhanced gated attention fusion module capable of adaptively learning inter-modal weights to achieve dynamic, synergistic integration at the feature level. A thorough investigation of the publicly accessible DEAP and MAHNOB-HCI datasets reveals that GBV-Net surpasses contemporary methods. Specifically, on the DEAP dataset, the model attained classification accuracies of 95.10% for Valence and 95.65% for Arousal, with F1-scores of 95.52% and 96.35%, respectively. On MAHNOB-HCI, the accuracies achieved were 97.28% for Valence and 97.73% for Arousal, with F1-scores of 97.50% and 97.74%, respectively. These experimental findings substantiate that GBV-Net effectively captures deep-level interactive information between multimodal signals, thereby improving emotion recognition accuracy.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

EmotionsFacial ExpressionHumansNeural Networks, ComputerElectroencephalographyElectrocardiographyAlgorithmsSignal Processing, Computer-Assisted

Stichwörter der Autor:innen

GBV-Netemotion recognitionfacial expressionsmultimodal fusionphysiological signals

Autor:innen und Institutionen

  • School of Information Engineering, Zhejiang Ocean University, Zhoushan 316022, China.

ORCID-Profile: Hongming Chen

Zitieren

APA

Yu, J., Ru, Y., Lei, B., & Chen, H. (2025). GBV-Net: Hierarchical Fusion of Facial Expressions and Physiological Signals for Multimodal Emotion Recognition. Sensors. https://doi.org/10.3390/s25206397

Vancouver

Yu J, Ru Y, Lei B, Chen H. GBV-Net: Hierarchical Fusion of Facial Expressions and Physiological Signals for Multimodal Emotion Recognition. Sensors. 2025. doi:10.3390/s25206397

BibTeX
@article{Yu2025,
  author = {Yu, Jiling and Ru, Yandong and Lei, Bangjun and Chen, Hongming},
  title = {GBV-Net: Hierarchical Fusion of Facial Expressions and Physiological Signals for Multimodal Emotion Recognition},
  journal = {Sensors},
  year = {2025},
  doi = {10.3390/s25206397}
}
Kostenlos starten →