EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Emotion Recognition Using EEG Signals and Audiovisual Features with Contrastive Learning

Ju-Hwan Lee iD, Jin-Young Kim iD, Hyoung-Gook Kim

Bioengineering · 03.10.2024 · 11 (10) · 997

DOI: 10.3390/bioengineering11100997 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Bioengineering, online erschienen am 03.10.2024. Verfasst von Ju-Hwan Lee und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Intelligent Electronics and Computer Engineering, Chonnam National University, 77 Yongbong-ro, Buk-gu, Gwangju 61186, Republic of Korea. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 90 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Research Foundation of Korea(NRF).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal emotion recognition has emerged as a promising approach to capture the complex nature of human emotions by integrating information from various sources such as physiological signals, visual behavioral cues, and audio-visual content. However, current methods often struggle with effectively processing redundant or conflicting information across modalities and may overlook implicit inter-modal correlations. To address these challenges, this paper presents a novel multimodal emotion recognition framework which integrates audio-visual features with viewers’ EEG data to enhance emotion classification accuracy. The proposed approach employs modality-specific encoders to extract spatiotemporal features, which are then aligned through contrastive learning to capture inter-modal relationships. Additionally, cross-modal attention mechanisms are incorporated for effective feature fusion across modalities. The framework, comprising pre-training, fine-tuning, and testing phases, is evaluated on multiple datasets of emotional responses. The experimental results demonstrate that the proposed multimodal approach, which combines audio-visual features with EEG data, is highly effective in recognizing emotions, highlighting its potential for advancing emotion recognition systems.

Schlagwörter

Stichwörter der Autor:innen

contrastive learningcross-attention mechanismemotion recognitionmultimodal learning

Autor:innen und Institutionen

  • Department of Intelligent Electronics and Computer Engineering, Chonnam National University, 77 Yongbong-ro, Buk-gu, Gwangju 61186, Republic of Korea.
  • Department of Electronic Convergence Engineering, Kwangwoon University, 20 Gwangun-ro, Nowon-gu, Seoul 01897, Republic of Korea.

ORCID-Profile: Ju-Hwan Lee, Jin-Young Kim

Förderung

  • National Research Foundation of Korea(NRF)

Zitieren

APA

Lee, J. H., Kim, J. Y., & Kim, H. G. (2024). Emotion Recognition Using EEG Signals and Audiovisual Features with Contrastive Learning. Bioengineering, 11(10), 997. https://doi.org/10.3390/bioengineering11100997

Vancouver

Lee JH, Kim JY, Kim HG. Emotion Recognition Using EEG Signals and Audiovisual Features with Contrastive Learning. Bioengineering. 2024;11(10):997. doi:10.3390/bioengineering11100997

BibTeX
@article{Lee2024,
  author = {Lee, Ju-Hwan and Kim, Jin-Young and Kim, Hyoung-Gook},
  title = {Emotion Recognition Using EEG Signals and Audiovisual Features with Contrastive Learning},
  journal = {Bioengineering},
  year = {2024},
  volume = {11},
  number = {10},
  pages = {997},
  doi = {10.3390/bioengineering11100997}
}
Kostenlos starten →