EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Self-Learning Multimodal Emotion Recognition Based on Multi-Scale Dilated Attention

Xiuli Du iD, Luyao Zhu

Brain Sciences · 25.03.2026 · 16 (4) · 350

DOI: 10.3390/brainsci16040350 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Brain Sciences, online erschienen am 25.03.2026. Verfasst von Xiuli Du und 1 weiteren Autor:innen. Beteiligte Einrichtung: School of Information Engineering, Dalian University, Dalian 116622, China. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Liaoning Provincial Education Department.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background/Objectives: Emotions can be recognized through external behavioral cues and internal physiological signals. Owing to the inherently complex psychological and physiological nature of emotions, models relying on a single modality often suffer from limited robustness. This study aims to improve emotion recognition performance by effectively integrating electroencephalogram (EEG) signals and facial expressions through a multimodal framework.

Methods

We propose a multimodal emotion recognition model that employs a Multi-Scale Dilated Attention Convolution (MSDAC) network tailored for facial expression recognition, integrates an EEG emotion recognition method based on three-dimensional features, and adopts a self-learning decision-level fusion strategy. MSDAC incorporates Multi-Scale Dilated Convolutions and a Dual-Branch Attention (D-BA) module to capture discontinuous facial action units. For EEG processing, raw signals are converted into a multidimensional time–frequency–spatial representation to preserve temporal, spectral, and spatial information. To overcome the limitations of traditional stitching or fixed-weight fusion approaches, a self-learning weight fusion mechanism is introduced at the decision level to adaptively adjust modality contributions.

Results

The facial analysis branch achieved average accuracies of 74.1% on FER2013, 99.69% on CK+, and 98.05% (valence)/96.15% (arousal) on DEAP. On the DEAP dataset, the complete multimodal model reached 98.66% accuracy for valence and 97.49% for arousal classification.

Conclusions

The proposed framework enhances emotion recognition by improving facial feature extraction and enabling adaptive multimodal fusion, demonstrating the effectiveness of combining EEG and facial information for robust emotion analysis.

Schlagwörter

Stichwörter der Autor:innen

EEGaction unitsdecision-level fusiondifferential entropyemotion recognitionfacial expressionsmulti-scale dilated convolutionmultimodal

Autor:innen und Institutionen

  • School of Information Engineering, Dalian University, Dalian 116622, China.
  • Communication and Network Laboratory, Dalian University, Dalian 116622, China.

ORCID-Profile: Xiuli Du

Förderung

  • Liaoning Provincial Education Department

Zitieren

APA

Du, X., & Zhu, L. (2026). Self-Learning Multimodal Emotion Recognition Based on Multi-Scale Dilated Attention. Brain Sciences, 16(4), 350. https://doi.org/10.3390/brainsci16040350

Vancouver

Du X, Zhu L. Self-Learning Multimodal Emotion Recognition Based on Multi-Scale Dilated Attention. Brain Sciences. 2026;16(4):350. doi:10.3390/brainsci16040350

BibTeX
@article{Du2026,
  author = {Du, Xiuli and Zhu, Luyao},
  title = {Self-Learning Multimodal Emotion Recognition Based on Multi-Scale Dilated Attention},
  journal = {Brain Sciences},
  year = {2026},
  volume = {16},
  number = {4},
  pages = {350},
  doi = {10.3390/brainsci16040350}
}
Kostenlos starten →