EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Predicting the Arousal and Valence Values of Emotional States Using Learned, Predesigned, and Deep Visual Features

Itaf Omar Joudeh iD, Ana-Maria Cretu iD, Stéphane Bouchard iD

Sensors · 2024

DOI: 10.3390/s24134398 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, 2024. Verfasst von Itaf Omar Joudeh und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Engineering, University of Quebec in Outaouais, Gatineau, QC J8Y 3G5, Canada. Hauptthemen laut MEDLINE: Emotions, Arousal, Neural Networks, Computer. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Sciences and Engineering Research Council, Natural Sciences and Engineering Research Council of Canada (NSERC)’s Discovery.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The cognitive state of a person can be categorized using the circumplex model of emotional states, a continuous model of two dimensions: arousal and valence. The purpose of this research is to select a machine learning model(s) to be integrated into a virtual reality (VR) system that runs cognitive remediation exercises for people with mental health disorders. As such, the prediction of emotional states is essential to customize treatments for those individuals. We exploit the Remote Collaborative and Affective Interactions (RECOLA) database to predict arousal and valence values using machine learning techniques. RECOLA includes audio, video, and physiological recordings of interactions between human participants. To allow learners to focus on the most relevant data, features are extracted from raw data. Such features can be predesigned, learned, or extracted implicitly using deep learners. Our previous work on video recordings focused on predesigned and learned visual features. In this paper, we extend our work onto deep visual features. Our deep visual features are extracted using the MobileNet-v2 convolutional neural network (CNN) that we previously trained on RECOLA’s video frames of full/half faces. As the final purpose of our work is to integrate our solution into a practical VR application using head-mounted displays, we experimented with half faces as a proof of concept. The extracted deep features were then used to predict arousal and valence values via optimizable ensemble regression. We also fused the extracted visual features with the predesigned visual features and predicted arousal and valence values using the combined feature set. In an attempt to enhance our prediction performance, we further fused the predictions of the optimizable ensemble model with the predictions of the MobileNet-v2 model. After decision fusion, we achieved a root mean squared error (RMSE) of 0.1140, a Pearson’s correlation coefficient (PCC) of 0.8000, and a concordance correlation coefficient (CCC) of 0.7868 on arousal predictions. We achieved an RMSE of 0.0790, a PCC of 0.7904, and a CCC of 0.7645 on valence predictions.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

EmotionsArousalNeural Networks, ComputerHumansMachine LearningVirtual RealityFemaleMaleDeep LearningAdult

Stichwörter der Autor:innen

cognitive/emotional statemachine learningregressionvisual features

Autor:innen und Institutionen

  • Department of Computer Science and Engineering, University of Quebec in Outaouais, Gatineau, QC J8Y 3G5, Canada.
  • Department of Psychoeducation and Psychology, University of Quebec in Outaouais, Gatineau, QC J8X 3X7, Canada.

ORCID-Profile: Itaf Omar Joudeh, Ana-Maria Cretu, Stéphane Bouchard

Förderung

  • Natural Sciences and Engineering Research Council
  • Natural Sciences and Engineering Research Council of Canada (NSERC)’s Discovery

Zitieren

APA

Joudeh, I. O., Cretu, A. M., & Bouchard, S. (2024). Predicting the Arousal and Valence Values of Emotional States Using Learned, Predesigned, and Deep Visual Features. Sensors. https://doi.org/10.3390/s24134398

Vancouver

Joudeh IO, Cretu AM, Bouchard S. Predicting the Arousal and Valence Values of Emotional States Using Learned, Predesigned, and Deep Visual Features. Sensors. 2024. doi:10.3390/s24134398

BibTeX
@article{Joudeh2024,
  author = {Joudeh, Itaf Omar and Cretu, Ana-Maria and Bouchard, Stéphane},
  title = {Predicting the Arousal and Valence Values of Emotional States Using Learned, Predesigned, and Deep Visual Features},
  journal = {Sensors},
  year = {2024},
  doi = {10.3390/s24134398}
}
Kostenlos starten →