Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures
Applied Sciences · 15.05.2024 · 14 (10) · 4199
DOI: 10.3390/app14104199 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 15.05.2024. Verfasst von Fazliddin Makhmudov und 2 weiteren Autor:innen. Die Arbeit wurde bisher 38-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Korea Agency for Technology and Standards.
Abstract (Originalsprache)
Emotion detection holds significant importance in facilitating human–computer interaction, enhancing the depth of engagement. By integrating this capability, we pave the way for forthcoming AI technologies to possess a blend of cognitive and emotional understanding, bridging the divide between machine functionality and human emotional complexity. This progress has the potential to reshape how machines perceive and respond to human emotions, ushering in an era of empathetic and intuitive artificial systems. The primary research challenge involves developing models that can accurately interpret and analyze emotions from both auditory and textual data, whereby auditory data require optimizing CNNs to detect subtle and intense emotional fluctuations in speech, and textual data necessitate access to large, diverse datasets to effectively capture nuanced emotional cues in written language. This paper introduces a novel approach to multimodal emotion recognition, seamlessly integrating speech and text modalities to accurately infer emotional states. Employing CNNs, we meticulously analyze speech using Mel spectrograms, while a BERT-based model processes the textual component, leveraging its bidirectional layers to enable profound semantic comprehension. The outputs from both modalities are combined using an attention-based fusion mechanism that optimally weighs their contributions. The proposed method here undergoes meticulous testing on two distinct datasets: Carnegie Mellon University’s Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset and the Multimodal Emotion Lines Dataset (MELD). The results demonstrate superior efficacy compared to existing frameworks, achieving an accuracy of 88.4% and an F1-score of 87.9% on the CMU-MOSEI dataset, and a notable weighted accuracy (WA) of 67.81% and a weighted F1 (WF1) score of 66.32% on the MELD dataset. This comprehensive system offers precise emotion detection and introduces several significant advancements in the field.
Förderung
- Korea Agency for Technology and Standards
Zitieren
APA
Makhmudov, F., Kultimuratov, A., & Cho, Y. (2024). Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures. Applied Sciences, 14(10), 4199. https://doi.org/10.3390/app14104199
Vancouver
Makhmudov F, Kultimuratov A, Cho Y. Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures. Applied Sciences. 2024;14(10):4199. doi:10.3390/app14104199
BibTeX
@article{Makhmudov2024,
author = {Makhmudov, Fazliddin and Kultimuratov, Alpamis and Cho, Young-Im},
title = {Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures},
journal = {Applied Sciences},
year = {2024},
volume = {14},
number = {10},
pages = {4199},
doi = {10.3390/app14104199}
}
EUVIMED