A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Entropy · 12.10.2023 · 25 (10) · 1440
DOI: 10.3390/e25101440 ↗Kurz gefasst
Übersichtsarbeit in der Zeitschrift Entropy, online erschienen am 12.10.2023. Verfasst von Hailun Lian und 5 weiteren Autor:innen. Beteiligte Einrichtung: Key Laboratory of Child Development and Learning Science (Ministry of Education), Southeast University, Nanjing 210000, China. Die Arbeit wurde bisher 231-mal zitiert (Crossref) und nennt 133 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key R & D Project, Zhishan Young Scholarship of Southeast University, Postdoctoral Scientific Research Foundation of Southeast University.
Abstract (Originalsprache)
Multimodal emotion recognition (MER) refers to the identification and understanding of human emotional states by combining different signals, including—but not limited to—text, speech, and face cues. MER plays a crucial role in the human–computer interaction (HCI) domain. With the recent progression of deep learning technologies and the increasing availability of multimodal datasets, the MER domain has witnessed considerable development, resulting in numerous significant research breakthroughs. However, a conspicuous absence of thorough and focused reviews on these deep learning-based MER achievements is observed. This survey aims to bridge this gap by providing a comprehensive overview of the recent advancements in MER based on deep learning. For an orderly exposition, this paper first outlines a meticulous analysis of the current multimodal datasets, emphasizing their advantages and constraints. Subsequently, we thoroughly scrutinize diverse methods for multimodal emotional feature extraction, highlighting the merits and demerits of each method. Moreover, we perform an exhaustive analysis of various MER algorithms, with particular focus on the model-agnostic fusion methods (including early fusion, late fusion, and hybrid fusion) and fusion based on intermediate layers of deep models (encompassing simple concatenation fusion, utterance-level interaction fusion, and fine-grained interaction fusion). We assess the strengths and weaknesses of these fusion strategies, providing guidance to researchers to help them select the most suitable techniques for their studies. In summary, this survey aims to provide a thorough and insightful review of the field of deep learning-based MER. It is intended as a valuable guide to aid researchers in furthering the evolution of this dynamic and impactful field.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Key Laboratory of Child Development and Learning Science (Ministry of Education), Southeast University, Nanjing 210000, China.
- School of Information Science and Engineering, Southeast University, Nanjing 210000, China.
- School of Biological Science and Medical Engineering, Southeast University, Nanjing 210000, China.
ORCID-Profile: Yan Zhao
Förderung
- National Key R & D Project
- Zhishan Young Scholarship of Southeast University
- Postdoctoral Scientific Research Foundation of Southeast University
- Jiangsu Province Excellent Postdoctoral Program
Zitieren
APA
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023). A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face. Entropy, 25(10), 1440. https://doi.org/10.3390/e25101440
Vancouver
Lian H, Lu C, Li S, Zhao Y, Tang C, Zong Y. A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face. Entropy. 2023;25(10):1440. doi:10.3390/e25101440
BibTeX
@article{Lian2023,
author = {Lian, Hailun and Lu, Cheng and Li, Sunan and Zhao, Yan and Tang, Chuangao and Zong, Yuan},
title = {A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face},
journal = {Entropy},
year = {2023},
volume = {25},
number = {10},
pages = {1440},
doi = {10.3390/e25101440}
}
EUVIMED