EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning

Shlomi Talala iD, Shaul Shvimmer, Rotem Simhon, Michael Gilead, Yitzhak Yitzhaky iD

Sensors · 2024

DOI: 10.3390/s24082620 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, 2024. Verfasst von Shlomi Talala und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Electro-Optics and Photonics Engineering, School of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Beer Sheva…. Hauptthemen laut MEDLINE: Deep Learning, Emotions, Facial Expression, Heart Rate. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: the JOY Neuro-wellness Grant.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Most human emotion recognition methods largely depend on classifying stereotypical facial expressions that represent emotions. However, such facial expressions do not necessarily correspond to actual emotional states and may correspond to communicative intentions. In other cases, emotions are hidden, cannot be expressed, or may have lower arousal manifested by less pronounced facial expressions, as may occur during passive video viewing. This study improves an emotion classification approach developed in a previous study, which classifies emotions remotely without relying on stereotypical facial expressions or contact-based methods, using short facial video data. In this approach, we desire to remotely sense transdermal cardiovascular spatiotemporal facial patterns associated with different emotional states and analyze this data via machine learning. In this paper, we propose several improvements, which include a better remote heart rate estimation via a preliminary skin segmentation, improvement of the heartbeat peaks and troughs detection process, and obtaining a better emotion classification accuracy by employing an appropriate deep learning classifier using an RGB camera input only with data. We used the dataset obtained in the previous study, which contains facial videos of 110 participants who passively viewed 150 short videos that elicited the following five emotion types: amusement, disgust, fear, sexual arousal, and no emotion, while three cameras with different wavelength sensitivities (visible spectrum, near-infrared, and longwave infrared) recorded them simultaneously. From the short facial videos, we extracted unique high-resolution spatiotemporal, physiologically affected features and examined them as input features with different deep-learning approaches. An EfficientNet-B0 model type was able to classify participants’ emotional states with an overall average accuracy of 47.36% using a single input spatiotemporal feature map obtained from a regular RGB camera.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Deep LearningEmotionsFacial ExpressionHeart RateHumansVideo RecordingImage Processing, Computer-AssistedFaceFemaleMale

Stichwörter der Autor:innen

camera-based PPGdeep learningemotion classificationpulsatile signalrPPGremote emotion recognition

Autor:innen und Institutionen

  • Department of Electro-Optics and Photonics Engineering, School of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Beer Sheva 84105, Israel.
  • School of Psychology, Tel Aviv University, Tel Aviv 39040, Israel.

ORCID-Profile: Shlomi Talala, Yitzhak Yitzhaky

Förderung

  • the JOY Neuro-wellness Grant

Zitieren

APA

Talala, S., Shvimmer, S., Simhon, R., Gilead, M., & Yitzhaky, Y. (2024). Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning. Sensors. https://doi.org/10.3390/s24082620

Vancouver

Talala S, Shvimmer S, Simhon R, Gilead M, Yitzhaky Y. Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning. Sensors. 2024. doi:10.3390/s24082620

BibTeX
@article{Talala2024,
  author = {Talala, Shlomi and Shvimmer, Shaul and Simhon, Rotem and Gilead, Michael and Yitzhaky, Yitzhak},
  title = {Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning},
  journal = {Sensors},
  year = {2024},
  doi = {10.3390/s24082620}
}
Kostenlos starten →