EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Multimodal Transformer-Based Framework for Emotion Analysis in Multilingual Video Content

Sehmus Yakut, Yusuf Taha Tuten, Eren Caglar, Mehmet S. Aktas

Computers · 01.02.2026 · 15 (2) · 77

DOI: 10.3390/computers15020077 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computers, online erschienen am 01.02.2026. Verfasst von Sehmus Yakut und 3 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 64 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This research addresses the challenge of inferring complex psychological states, including stress, fatigue, anxiety, cognitive load, and boredom, from facial expressions. We propose an interpretable, literature-informed emotion-weighting methodology that transforms the eight-emotion probability outputs of facial emotion recognition models into continuous estimates of these five psychological states using weights derived from the Valence–Arousal framework, providing a principled bridge between discrete emotion predictions and higher-level affective constructs. The proposed formulation is evaluated across six representative deep learning architectures—a baseline CNN (ResNet-50), a modern CNN (ConvNeXt), a hybrid attention-based model (DDAMFN), and three Transformer-based models (ViT, BEiT, and Swin). Our results demonstrate that strong performance on discrete FER tasks does not directly translate to consistent behavior in complex state inference; instead, architectures capable of preserving subtle and distributed affective cues yield more stable and interpretable state estimates, with DDAMFN and Vision Transformer models exhibiting the most consistent performance across the evaluated psychological states. These findings highlight the central role of the proposed emotion-weighting formulation and the importance of architecture selection beyond categorical accuracy in complex affective state analysis.

Zitieren

APA

Yakut, S., Tuten, Y. T., Caglar, E., & Aktas, M. S. (2026). A Multimodal Transformer-Based Framework for Emotion Analysis in Multilingual Video Content. Computers, 15(2), 77. https://doi.org/10.3390/computers15020077

Vancouver

Yakut S, Tuten YT, Caglar E, Aktas MS. A Multimodal Transformer-Based Framework for Emotion Analysis in Multilingual Video Content. Computers. 2026;15(2):77. doi:10.3390/computers15020077

BibTeX
@article{Yakut2026,
  author = {Yakut, Sehmus and Tuten, Yusuf Taha and Caglar, Eren and Aktas, Mehmet S.},
  title = {A Multimodal Transformer-Based Framework for Emotion Analysis in Multilingual Video Content},
  journal = {Computers},
  year = {2026},
  volume = {15},
  number = {2},
  pages = {77},
  doi = {10.3390/computers15020077}
}
Kostenlos starten →