EmoAsst: emotion recognition assistant via text-guided transfer learning on pre-trained visual and acoustic models
Frontiers in Computer Science · 09.04.2024 · 6
DOI: 10.3389/fcomp.2024.1304687 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Computer Science, online erschienen am 09.04.2024. Verfasst von Minxiao Wang und 1 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Southern Illinois University Carbondale.
Abstract (Originalsprache)
Children diagnosed with Autism Spectrum Disorder (ASD) often struggle to grasp social conventions and promptly recognize others' emotions. Recent advancements in the application of deep learning (DL) to emotion recognition are solidifying the role of AI-powered assistive technology in supporting autistic children. However, the cost of collecting and annotating large-scale high-quality human emotion data and the phenomenon of unbalanced performance on different modalities of data challenge DL-based emotion recognition. In response to these challenges, this paper explores transfer learning, wherein large pre-trained models like Contrastive Language-Image Pre-training (CLIP) and wav2vec 2.0 are fine-tuned to improve audio- and video-based emotion recognition with text- based guidance. In this work, we propose the EmoAsst framework, which includes a visual fusion module and emotion prompt fine-tuning for CLIP, in addition to leveraging CLIP's text encoder and supervised contrastive learning for audio-based emotion recognition on the wav2vec 2.0 model. In addition, a joint few-shot emotion classifier enhances the accuracy and offers great adaptability for real-world applications. The evaluation results on the MELD dataset highlight the outstanding performance of our methods, surpassing the majority of existing video and audio-based approaches. Notably, our research demonstrates the promising potential of the proposed text-based guidance techniques for improving video and audio-based Emotion Recognition and Classification (ERC).
Förderung
- Southern Illinois University Carbondale
Zitieren
APA
Wang, M., & Yang, N. (2024). EmoAsst: emotion recognition assistant via text-guided transfer learning on pre-trained visual and acoustic models. Frontiers in Computer Science, 6. https://doi.org/10.3389/fcomp.2024.1304687
Vancouver
Wang M, Yang N. EmoAsst: emotion recognition assistant via text-guided transfer learning on pre-trained visual and acoustic models. Frontiers in Computer Science. 2024;6. doi:10.3389/fcomp.2024.1304687
BibTeX
@article{Wang2024,
author = {Wang, Minxiao and Yang, Ning},
title = {EmoAsst: emotion recognition assistant via text-guided transfer learning on pre-trained visual and acoustic models},
journal = {Frontiers in Computer Science},
year = {2024},
volume = {6},
doi = {10.3389/fcomp.2024.1304687}
}
EUVIMED