EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Bimodal Emotion Recognition Approach through the Fusion of Electroencephalography and Facial Sequences

Farah Muhammad iD, Muhammad Hussain iD, Hatim Aboalsamh iD

Diagnostics · 04.03.2023 · 13 (5) · 977

DOI: 10.3390/diagnostics13050977 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Diagnostics, online erschienen am 04.03.2023. Verfasst von Farah Muhammad und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, College of Computer Science and Information, King Saud University, Riyadh 11451, Saudi Arabia. Die Arbeit wurde bisher 41-mal zitiert (Crossref) und nennt 89 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: the Deputyship for Research and Innovation, Ministry of Education in Saudi Arabia, Deputyship for Research and Innovation, Ministry of Education in Saudi Arabia.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In recent years, human–computer interaction (HCI) systems have become increasingly popular. Some of these systems demand particular approaches for discriminating actual emotions through the use of better multimodal methods. In this work, a deep canonical correlation analysis (DCCA) based multimodal emotion recognition method is presented through the fusion of electroencephalography (EEG) and facial video clips. A two-stage framework is implemented, where the first stage extracts relevant features for emotion recognition using a single modality, while the second stage merges the highly correlated features from the two modalities and performs classification. Convolutional neural network (CNN) based Resnet50 and 1D-CNN (1-Dimensional CNN) have been utilized to extract features from facial video clips and EEG modalities, respectively. A DCCA-based approach was used to fuse highly correlated features, and three basic human emotion categories (happy, neutral, and sad) were classified using the SoftMax classifier. The proposed approach was investigated based on the publicly available datasets called MAHNOB-HCI and DEAP. Experimental results revealed an average accuracy of 93.86% and 91.54% on the MAHNOB-HCI and DEAP datasets, respectively. The competitiveness of the proposed framework and the justification for exclusivity in achieving this accuracy were evaluated by comparison with existing work.

Schlagwörter

Stichwörter der Autor:innen

CNNDeep CCAHCIbimodalelectroencephalographyemotion recognitionfacial video clipsfeature level fusion

Autor:innen und Institutionen

  • Department of Computer Science, College of Computer Science and Information, King Saud University, Riyadh 11451, Saudi Arabia.

ORCID-Profile: Farah Muhammad, Muhammad Hussain, Hatim Aboalsamh

Förderung

  • the Deputyship for Research and Innovation, Ministry of Education in Saudi Arabia
  • Deputyship for Research and Innovation, Ministry of Education in Saudi Arabia

Zitieren

APA

Muhammad, F., Hussain, M., & Aboalsamh, H. (2023). A Bimodal Emotion Recognition Approach through the Fusion of Electroencephalography and Facial Sequences. Diagnostics, 13(5), 977. https://doi.org/10.3390/diagnostics13050977

Vancouver

Muhammad F, Hussain M, Aboalsamh H. A Bimodal Emotion Recognition Approach through the Fusion of Electroencephalography and Facial Sequences. Diagnostics. 2023;13(5):977. doi:10.3390/diagnostics13050977

BibTeX
@article{Muhammad2023,
  author = {Muhammad, Farah and Hussain, Muhammad and Aboalsamh, Hatim},
  title = {A Bimodal Emotion Recognition Approach through the Fusion of Electroencephalography and Facial Sequences},
  journal = {Diagnostics},
  year = {2023},
  volume = {13},
  number = {5},
  pages = {977},
  doi = {10.3390/diagnostics13050977}
}
Kostenlos starten →