Emotion Recognition Using EEG Signals and Audiovisual Features with Contrastive Learning
Bioengineering · 03.10.2024 · 11 (10) · 997
DOI: 10.3390/bioengineering11100997 ↗Kurz gefasst
Fachartikel in der Zeitschrift Bioengineering, online erschienen am 03.10.2024. Verfasst von Ju-Hwan Lee und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Intelligent Electronics and Computer Engineering, Chonnam National University, 77 Yongbong-ro, Buk-gu, Gwangju 61186, Republic of Korea. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 90 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Research Foundation of Korea(NRF).
Abstract (Originalsprache)
Multimodal emotion recognition has emerged as a promising approach to capture the complex nature of human emotions by integrating information from various sources such as physiological signals, visual behavioral cues, and audio-visual content. However, current methods often struggle with effectively processing redundant or conflicting information across modalities and may overlook implicit inter-modal correlations. To address these challenges, this paper presents a novel multimodal emotion recognition framework which integrates audio-visual features with viewers’ EEG data to enhance emotion classification accuracy. The proposed approach employs modality-specific encoders to extract spatiotemporal features, which are then aligned through contrastive learning to capture inter-modal relationships. Additionally, cross-modal attention mechanisms are incorporated for effective feature fusion across modalities. The framework, comprising pre-training, fine-tuning, and testing phases, is evaluated on multiple datasets of emotional responses. The experimental results demonstrate that the proposed multimodal approach, which combines audio-visual features with EEG data, is highly effective in recognizing emotions, highlighting its potential for advancing emotion recognition systems.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Intelligent Electronics and Computer Engineering, Chonnam National University, 77 Yongbong-ro, Buk-gu, Gwangju 61186, Republic of Korea.
- Department of Electronic Convergence Engineering, Kwangwoon University, 20 Gwangun-ro, Nowon-gu, Seoul 01897, Republic of Korea.
ORCID-Profile: Ju-Hwan Lee, Jin-Young Kim
Förderung
- National Research Foundation of Korea(NRF)
Zitieren
APA
Lee, J. H., Kim, J. Y., & Kim, H. G. (2024). Emotion Recognition Using EEG Signals and Audiovisual Features with Contrastive Learning. Bioengineering, 11(10), 997. https://doi.org/10.3390/bioengineering11100997
Vancouver
Lee JH, Kim JY, Kim HG. Emotion Recognition Using EEG Signals and Audiovisual Features with Contrastive Learning. Bioengineering. 2024;11(10):997. doi:10.3390/bioengineering11100997
BibTeX
@article{Lee2024,
author = {Lee, Ju-Hwan and Kim, Jin-Young and Kim, Hyoung-Gook},
title = {Emotion Recognition Using EEG Signals and Audiovisual Features with Contrastive Learning},
journal = {Bioengineering},
year = {2024},
volume = {11},
number = {10},
pages = {997},
doi = {10.3390/bioengineering11100997}
}
EUVIMED