Multimodal emotion recognition based on a fusion of audiovisual information with temporal dynamics
Multimedia Tools and Applications · 18.09.2024 · 84 (23) · 27327-27343
DOI: 10.1007/s11042-024-20227-6 ↗Kurz gefasst
Fachartikel in der Zeitschrift Multimedia Tools and Applications, online erschienen am 18.09.2024. Verfasst von José Salas-Cáceres und 3 weiteren Autor:innen. Die Arbeit wurde bisher 49-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Universidad de las Palmas de Gran Canaria.
Abstract (Originalsprache)
In the Human-Machine Interactions (HMI) landscape, understanding user emotions is pivotal for elevating user experiences. This paper explores Facial Expression Recognition (FER) within HMI, employing a distinctive multimodal approach that integrates visual and auditory information. Recognizing the dynamic nature of HMI, where situations evolve, this study emphasizes continuous emotion analysis. This work assesses various fusion strategies that involve the addition to the main network of different architectures, such as autoencoders (AE) or an Embracement module, to combine the information of multiple biometric cues. In addition to the multimodal approach, this paper introduces a new architecture that prioritizes temporal dynamics by incorporating Long Short-Term Memory (LSTM) networks. The final proposal, which integrates different multimodal approaches with the temporal focus capabilities of the LSTM architecture, was tested across three public datasets: RAVDESS, SAVEE, and CREMA-D. It showcased state-of-the-art accuracy of 88.11%, 86.75%, and 80.27%, respectively, and outperformed other existing approaches.
Förderung
- Universidad de las Palmas de Gran Canaria
Zitieren
APA
Salas-Cáceres, J., Lorenzo-Navarro, J., Freire-Obregón, D., & Castrillón-Santana, M. (2024). Multimodal emotion recognition based on a fusion of audiovisual information with temporal dynamics. Multimedia Tools and Applications, 84(23), 27327-27343. https://doi.org/10.1007/s11042-024-20227-6
Vancouver
Salas-Cáceres J, Lorenzo-Navarro J, Freire-Obregón D, Castrillón-Santana M. Multimodal emotion recognition based on a fusion of audiovisual information with temporal dynamics. Multimedia Tools and Applications. 2024;84(23):27327-27343. doi:10.1007/s11042-024-20227-6
BibTeX
@article{SalasCceres2024,
author = {Salas-Cáceres, José and Lorenzo-Navarro, Javier and Freire-Obregón, David and Castrillón-Santana, Modesto},
title = {Multimodal emotion recognition based on a fusion of audiovisual information with temporal dynamics},
journal = {Multimedia Tools and Applications},
year = {2024},
volume = {84},
number = {23},
pages = {27327--27343},
doi = {10.1007/s11042-024-20227-6}
}
EUVIMED