AVaTER: Fusing Audio, Visual, and Textual Modalities Using Cross-Modal Attention for Emotion Recognition
Sensors · 10.09.2024 · 24 (18) · 5862
DOI: 10.3390/s24185862 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 10.09.2024. Verfasst von Avishek Das und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Engineering, Chittagong University of Engineering and Technology, Chittagong 4349, Bangladesh. Hauptthemen laut MEDLINE: Emotions. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Directorate of Research and Extension (DRE), Chittagong University of Engineering & Technology (CUET).
Abstract (Originalsprache)
Multimodal emotion classification (MEC) involves analyzing and identifying human emotions by integrating data from multiple sources, such as audio, video, and text. This approach leverages the complementary strengths of each modality to enhance the accuracy and robustness of emotion recognition systems. However, one significant challenge is effectively integrating these diverse data sources, each with unique characteristics and levels of noise. Additionally, the scarcity of large, annotated multimodal datasets in Bangla limits the training and evaluation of models. In this work, we unveiled a pioneering multimodal Bangla dataset, MAViT-Bangla (Multimodal Audio Video Text Bangla dataset). This dataset, comprising 1002 samples across audio, video, and text modalities, is a unique resource for emotion recognition studies in the Bangla language. It features emotional categories such as anger, fear, joy, and sadness, providing a comprehensive platform for research. Additionally, we developed a framework for audio, video and textual emotion recognition (i.e., AVaTER) that employs a cross-modal attention mechanism among unimodal features. This mechanism fosters the interaction and fusion of features from different modalities, enhancing the model’s ability to capture nuanced emotional cues. The effectiveness of this approach was demonstrated by achieving an F1-score of 0.64, a significant improvement over unimodal methods.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science and Engineering, Chittagong University of Engineering and Technology, Chittagong 4349, Bangladesh.
- School of Computing, Engineering and Intelligent Systems, Ulster University, Belfast BT15 1AP, UK.
- School of Computing and Information Systems, Faculty of Science and Technology, Athabasca University, Athabasca, AB T9S 3A3, Canada.
ORCID-Profile: Avishek Das, Moumita Sen Sarma, Mohammed Moshiul Hoque, Nazmul Siddique, M Ali Akber Dewan
Förderung
- Directorate of Research and Extension (DRE), Chittagong University of Engineering & Technology (CUET)
Zitieren
APA
Das, A., Sarma, M. S., Hoque, M. M., Siddique, N., & Dewan, M. A. A. (2024). AVaTER: Fusing Audio, Visual, and Textual Modalities Using Cross-Modal Attention for Emotion Recognition. Sensors, 24(18), 5862. https://doi.org/10.3390/s24185862
Vancouver
Das A, Sarma MS, Hoque MM, Siddique N, Dewan MAA. AVaTER: Fusing Audio, Visual, and Textual Modalities Using Cross-Modal Attention for Emotion Recognition. Sensors. 2024;24(18):5862. doi:10.3390/s24185862
BibTeX
@article{Das2024,
author = {Das, Avishek and Sarma, Moumita Sen and Hoque, Mohammed Moshiul and Siddique, Nazmul and Dewan, M Ali Akber},
title = {AVaTER: Fusing Audio, Visual, and Textual Modalities Using Cross-Modal Attention for Emotion Recognition},
journal = {Sensors},
year = {2024},
volume = {24},
number = {18},
pages = {5862},
doi = {10.3390/s24185862}
}
EUVIMED