EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

AVaTER: Fusing Audio, Visual, and Textual Modalities Using Cross-Modal Attention for Emotion Recognition

Avishek Das iD, Moumita Sen Sarma iD, Mohammed Moshiul Hoque iD, Nazmul Siddique iD, M Ali Akber Dewan iD

Sensors · 10.09.2024 · 24 (18) · 5862

DOI: 10.3390/s24185862 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 10.09.2024. Verfasst von Avishek Das und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Engineering, Chittagong University of Engineering and Technology, Chittagong 4349, Bangladesh. Hauptthemen laut MEDLINE: Emotions. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Directorate of Research and Extension (DRE), Chittagong University of Engineering & Technology (CUET).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal emotion classification (MEC) involves analyzing and identifying human emotions by integrating data from multiple sources, such as audio, video, and text. This approach leverages the complementary strengths of each modality to enhance the accuracy and robustness of emotion recognition systems. However, one significant challenge is effectively integrating these diverse data sources, each with unique characteristics and levels of noise. Additionally, the scarcity of large, annotated multimodal datasets in Bangla limits the training and evaluation of models. In this work, we unveiled a pioneering multimodal Bangla dataset, MAViT-Bangla (Multimodal Audio Video Text Bangla dataset). This dataset, comprising 1002 samples across audio, video, and text modalities, is a unique resource for emotion recognition studies in the Bangla language. It features emotional categories such as anger, fear, joy, and sadness, providing a comprehensive platform for research. Additionally, we developed a framework for audio, video and textual emotion recognition (i.e., AVaTER) that employs a cross-modal attention mechanism among unimodal features. This mechanism fosters the interaction and fusion of features from different modalities, enhancing the model’s ability to capture nuanced emotional cues. The effectiveness of this approach was demonstrated by achieving an F1-score of 0.64, a significant improvement over unimodal methods.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

EmotionsHumansVideo RecordingAttention

Stichwörter der Autor:innen

cross-modal attentionmultimodal datasetmultimodal emotion recognitionnatural language processingtransformers

Autor:innen und Institutionen

  • Department of Computer Science and Engineering, Chittagong University of Engineering and Technology, Chittagong 4349, Bangladesh.
  • School of Computing, Engineering and Intelligent Systems, Ulster University, Belfast BT15 1AP, UK.
  • School of Computing and Information Systems, Faculty of Science and Technology, Athabasca University, Athabasca, AB T9S 3A3, Canada.

ORCID-Profile: Avishek Das, Moumita Sen Sarma, Mohammed Moshiul Hoque, Nazmul Siddique, M Ali Akber Dewan

Förderung

  • Directorate of Research and Extension (DRE), Chittagong University of Engineering & Technology (CUET)

Zitieren

APA

Das, A., Sarma, M. S., Hoque, M. M., Siddique, N., & Dewan, M. A. A. (2024). AVaTER: Fusing Audio, Visual, and Textual Modalities Using Cross-Modal Attention for Emotion Recognition. Sensors, 24(18), 5862. https://doi.org/10.3390/s24185862

Vancouver

Das A, Sarma MS, Hoque MM, Siddique N, Dewan MAA. AVaTER: Fusing Audio, Visual, and Textual Modalities Using Cross-Modal Attention for Emotion Recognition. Sensors. 2024;24(18):5862. doi:10.3390/s24185862

BibTeX
@article{Das2024,
  author = {Das, Avishek and Sarma, Moumita Sen and Hoque, Mohammed Moshiul and Siddique, Nazmul and Dewan, M Ali Akber},
  title = {AVaTER: Fusing Audio, Visual, and Textual Modalities Using Cross-Modal Attention for Emotion Recognition},
  journal = {Sensors},
  year = {2024},
  volume = {24},
  number = {18},
  pages = {5862},
  doi = {10.3390/s24185862}
}
Kostenlos starten →