EUVIMEDObjevte EUVIMED
VergleichsstudieOpen access

Facial Emotion Recognition of 16 Distinct Emotions From Smartphone Videos: Comparative Study of Machine Learning and Human Performance

Marie Keinert iD, Simon Pistrosch iD, Adria Mallol-Ragolta iD, Björn W Schuller iD, Matthias Berking iD

Journal of Medical Internet Research · 2025-07-02 · 27 · e68942

DOI: 10.2196/68942 ↗

At a glance

Vergleichsstudie in Journal of Medical Internet Research, published online on 2025-07-02. Written by Marie Keinert and 4 co-authors. Institution: Department of Clinical Psychology and Psychotherapy, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany. Major MEDLINE topics: Smartphone, Emotions, Facial Expression, Machine Learning. Cited 16 times so far (Crossref); 68 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

The development of automatic emotion recognition models from smartphone videos is a crucial step toward the dissemination of psychotherapeutic app interventions that encourage emotional expressions. Existing models focus mainly on the 6 basic emotions while neglecting other therapeutically relevant emotions. To support this research, we introduce the novel Stress Reduction Training Through the Recognition of Emotions Wizard-of-Oz (STREs WoZ) dataset, which contains facial videos of 16 distinct, therapeutically relevant emotions.

Objective

This study aimed to develop deep learning–based automatic facial emotion recognition (FER) models for binary (positive vs negative) and multiclass emotion classification tasks, assess the models’ performance, and validate them by comparing the models with human observers.

Methods

The STREs WoZ dataset contains 14,412 facial videos of 63 individuals displaying the 16 emotions. The selfie-style videos were recorded during a stress reduction training using front-facing smartphone cameras in a nonconstrained laboratory setting. Automatic FER models using both appearance and deep-learned features for binary and multiclass emotion classification were trained on the STREs WoZ dataset. The appearance features were based on the Facial Action Coding System and extracted with OpenFace. The deep-learned features were obtained through a ResNet50 model. For our deep learning models, we used the appearance features, the deep-learned features, and their concatenation as inputs. We used 3 recurrent neural network (RNN)–based architectures: RNN-convolution, RNN-attention, and RNN-average networks. For validation, 3 human observers were also trained in binary and multiclass emotion recognition. A test set of 3018 facial emotion videos of the 16 emotions was completed by both the automatic FER model and human observers. The performance was assessed with unweighted average recall (UAR) and accuracy.

Results

Models using appearance features outperformed those using deep-learned features, as well as models combining both feature types in both tasks, with the attention network using appearance features emerging as the best-performing model. The attention network achieved a UAR of 92.9% in the binary classification task, and accuracy values ranged from 59.0% to 90.0% in the multiclass classification task. Human performance was comparable to that of the automatic FER model in the binary classification task, with a UAR of 91.0%, and superior in the multiclass classification task, with accuracy values ranging from 87.4% to 99.8%.

Conclusions

Future studies are needed to enhance the performance of automatic FER models for practical use in psychotherapeutic apps. Nevertheless, this study represents an important first step toward advancing emotion-focused psychotherapeutic interventions via smartphone apps.

Subjects

MeSH terms (MEDLINE)

SmartphoneEmotionsFacial ExpressionMachine LearningVideo RecordingHumansDeep LearningFemaleMaleAdultNeural Networks, Computer

Author keywords

16 distinct emotionsdeep learningfacial emotion recognitionsmartphone videovalidation

Authors and institutions

  • Department of Clinical Psychology and Psychotherapy, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany.
  • Chair of Health Informatics, Technical University of Munich, Munich, Germany.
  • Munich Center for Machine Learning, Munich, Germany.
  • Munich Data Science Institute, Munich, Germany.
  • Group on Language, Audio, & Music, Imperial College London, London, United Kingdom.

ORCID profiles: Marie Keinert, Simon Pistrosch, Adria Mallol-Ragolta, Björn W Schuller, Matthias Berking

Cite

APA

Keinert, M., Pistrosch, S., Mallol-Ragolta, A., Schuller, B. W., & Berking, M. (2025). Facial Emotion Recognition of 16 Distinct Emotions From Smartphone Videos: Comparative Study of Machine Learning and Human Performance. Journal of Medical Internet Research, 27, e68942. https://doi.org/10.2196/68942

Vancouver

Keinert M, Pistrosch S, Mallol-Ragolta A, Schuller BW, Berking M. Facial Emotion Recognition of 16 Distinct Emotions From Smartphone Videos: Comparative Study of Machine Learning and Human Performance. Journal of Medical Internet Research. 2025;27:e68942. doi:10.2196/68942

BibTeX
@article{Keinert2025,
  author = {Keinert, Marie and Pistrosch, Simon and Mallol-Ragolta, Adria and Schuller, Björn W and Berking, Matthias},
  title = {Facial Emotion Recognition of 16 Distinct Emotions From Smartphone Videos: Comparative Study of Machine Learning and Human Performance},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e68942},
  doi = {10.2196/68942}
}
Start for free →