EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A gated leaky integrate-and-fire spiking neural network based on attention mechanism for multi-modal emotion recognition

Guoming Chen, Zhuoxian Qian, Shuang Qiu, Dong Zhang, Ruqi Zhou

Digital Signal Processing · 2025-10 · 165 · 105322

DOI: 10.1016/j.dsp.2025.105322 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Digital Signal Processing, 2025-10. Verfasst von Guoming Chen und 4 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 89 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Guangzhou Municipal Science and Technology Program key projects, National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multi-modal emotion recognition is a key research area in human-computer interaction. It involves processing heterogeneous multi-modal signals, which present challenges in signal alignment while aiming to enhance accuracy and reduce computational complexity. To address these challenges, we apply swarm decomposition to EEG signals to reduce noise and extract Short-Time Fourier Transform features. Heatmap features are then derived from these signals, as well as from other non-physiological signals such as facial expressions, voice, and text. These features from various sources are aligned using Discrete Wavelet Transform. We propose a Gated Leaky Integrate-and-Fire Spiking Convolutional Vision Transformer(GLIFCVT) framework for multimodal emotion recognition. This framework utilizes visual features as the primary modality and incorporates a spiking gated attention mechanism to enhance multimodal fusion and classification. In addition, we propose a novel loss function that integrates Focal and Dice losses to address class imbalance. Experiments demonstrate our proposed model consistently outperforms state-of-the-art methods in both accuracy and energy efficiency.

Förderung

  • Guangzhou Municipal Science and Technology Program key projects
  • National Natural Science Foundation of China

Zitieren

APA

Chen, G., Qian, Z., Qiu, S., Zhang, D., & Zhou, R. (2025). A gated leaky integrate-and-fire spiking neural network based on attention mechanism for multi-modal emotion recognition. Digital Signal Processing, 165, 105322. https://doi.org/10.1016/j.dsp.2025.105322

Vancouver

Chen G, Qian Z, Qiu S, Zhang D, Zhou R. A gated leaky integrate-and-fire spiking neural network based on attention mechanism for multi-modal emotion recognition. Digital Signal Processing. 2025;165:105322. doi:10.1016/j.dsp.2025.105322

BibTeX
@article{Chen2025,
  author = {Chen, Guoming and Qian, Zhuoxian and Qiu, Shuang and Zhang, Dong and Zhou, Ruqi},
  title = {A gated leaky integrate-and-fire spiking neural network based on attention mechanism for multi-modal emotion recognition},
  journal = {Digital Signal Processing},
  year = {2025},
  volume = {165},
  pages = {105322},
  doi = {10.1016/j.dsp.2025.105322}
}
Kostenlos starten →