EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Multimodal emotion recognition based on a fusion of audiovisual information with temporal dynamics

José Salas-Cáceres, Javier Lorenzo-Navarro, David Freire-Obregón, Modesto Castrillón-Santana

Multimedia Tools and Applications · 18.09.2024 · 84 (23) · 27327-27343

DOI: 10.1007/s11042-024-20227-6 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Multimedia Tools and Applications, online erschienen am 18.09.2024. Verfasst von José Salas-Cáceres und 3 weiteren Autor:innen. Die Arbeit wurde bisher 49-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Universidad de las Palmas de Gran Canaria.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In the Human-Machine Interactions (HMI) landscape, understanding user emotions is pivotal for elevating user experiences. This paper explores Facial Expression Recognition (FER) within HMI, employing a distinctive multimodal approach that integrates visual and auditory information. Recognizing the dynamic nature of HMI, where situations evolve, this study emphasizes continuous emotion analysis. This work assesses various fusion strategies that involve the addition to the main network of different architectures, such as autoencoders (AE) or an Embracement module, to combine the information of multiple biometric cues. In addition to the multimodal approach, this paper introduces a new architecture that prioritizes temporal dynamics by incorporating Long Short-Term Memory (LSTM) networks. The final proposal, which integrates different multimodal approaches with the temporal focus capabilities of the LSTM architecture, was tested across three public datasets: RAVDESS, SAVEE, and CREMA-D. It showcased state-of-the-art accuracy of 88.11%, 86.75%, and 80.27%, respectively, and outperformed other existing approaches.

Förderung

  • Universidad de las Palmas de Gran Canaria

Zitieren

APA

Salas-Cáceres, J., Lorenzo-Navarro, J., Freire-Obregón, D., & Castrillón-Santana, M. (2024). Multimodal emotion recognition based on a fusion of audiovisual information with temporal dynamics. Multimedia Tools and Applications, 84(23), 27327-27343. https://doi.org/10.1007/s11042-024-20227-6

Vancouver

Salas-Cáceres J, Lorenzo-Navarro J, Freire-Obregón D, Castrillón-Santana M. Multimodal emotion recognition based on a fusion of audiovisual information with temporal dynamics. Multimedia Tools and Applications. 2024;84(23):27327-27343. doi:10.1007/s11042-024-20227-6

BibTeX
@article{SalasCceres2024,
  author = {Salas-Cáceres, José and Lorenzo-Navarro, Javier and Freire-Obregón, David and Castrillón-Santana, Modesto},
  title = {Multimodal emotion recognition based on a fusion of audiovisual information with temporal dynamics},
  journal = {Multimedia Tools and Applications},
  year = {2024},
  volume = {84},
  number = {23},
  pages = {27327--27343},
  doi = {10.1007/s11042-024-20227-6}
}
Kostenlos starten →