Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features
Sensors · 09.06.2023 · 23 (12) · 5475
DOI: 10.3390/s23125475 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 09.06.2023. Verfasst von Dilnoza Mamieva und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Engineering, Gachon University, Seongnam-si 13120, Republic of Korea. Hauptthemen laut MEDLINE: Speech, Emotions. Die Arbeit wurde bisher 127-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: GRRC program of Gyeonggi province.
Abstract (Originalsprache)
Methods for detecting emotions that employ many modalities at the same time have been found to be more accurate and resilient than those that rely on a single sense. This is due to the fact that sentiments may be conveyed in a wide range of modalities, each of which offers a different and complementary window into the thoughts and emotions of the speaker. In this way, a more complete picture of a person’s emotional state may emerge through the fusion and analysis of data from several modalities. The research suggests a new attention-based approach to multimodal emotion recognition. This technique integrates facial and speech features that have been extracted by independent encoders in order to pick the aspects that are the most informative. It increases the system’s accuracy by processing speech and facial features of various sizes and focuses on the most useful bits of input. A more comprehensive representation of facial expressions is extracted by the use of both low- and high-level facial features. These modalities are combined using a fusion network to create a multimodal feature vector which is then fed to a classification layer for emotion recognition. The developed system is evaluated on two datasets, IEMOCAP and CMU-MOSEI, and shows superior performance compared to existing models, achieving a weighted accuracy WA of 74.6% and an F1 score of 66.1% on the IEMOCAP dataset and a WA of 80.7% and F1 score of 73.7% on the CMU-MOSEI dataset.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Engineering, Gachon University, Seongnam-si 13120, Republic of Korea.
- Department of AI. Software, Gachon University, Seongnam-si 13120, Republic of Korea.
- Department of Artificial Intelligence, Tashkent State University of Economics, Tashkent 100066, Uzbekistan.
ORCID-Profile: Akmalbek Bobomirzaevich Abdusalomov
Förderung
- GRRC program of Gyeonggi province
Zitieren
APA
Mamieva, D., Abdusalomov, A. B., Kutlimuratov, A., Muminov, B., & Whangbo, T. K. (2023). Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features. Sensors, 23(12), 5475. https://doi.org/10.3390/s23125475
Vancouver
Mamieva D, Abdusalomov AB, Kutlimuratov A, Muminov B, Whangbo TK. Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features. Sensors. 2023;23(12):5475. doi:10.3390/s23125475
BibTeX
@article{Mamieva2023,
author = {Mamieva, Dilnoza and Abdusalomov, Akmalbek Bobomirzaevich and Kutlimuratov, Alpamis and Muminov, Bahodir and Whangbo, Taeg Keun},
title = {Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features},
journal = {Sensors},
year = {2023},
volume = {23},
number = {12},
pages = {5475},
doi = {10.3390/s23125475}
}
EUVIMED