EUVIMEDEUVIMED entdecken
Crossref

Multimodal Emotion Recognition Using Bi-LG-GCN for MELD Dataset

Hussein Farooq Tayeb Alsaadawı, Resul Daş

Balkan Journal of Electrical and Computer Engineering · 01.03.2024 · 12 (1) · 36-46

DOI: 10.17694/bajece.1372107 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Balkan Journal of Electrical and Computer Engineering, online erschienen am 01.03.2024. Verfasst von Hussein Farooq Tayeb Alsaadawı und 1 weiteren Autor:innen. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 41 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Emotion recognition using multimodal data is a widely adopted approach due to its potential to enhance human interactions and various applications. By leveraging multimodal data for emotion recognition, the quality of human interactions can be significantly improved. We present the Multimodal Emotion Lines Dataset (MELD) and a novel method for multimodal emotion recognition using a bi-lateral gradient graph neural network (Bi-LG-GNN) and feature extraction and pre-processing. The multimodal dataset uses fine-grained emotion labeling for textual, audio, and visual modalities. This work aims to identify affective computing states successfully concealed in the textual and audio data for emotion recognition and sentiment analysis. We use pre-processing techniques to improve the quality and consistency of the data to increase the dataset’s usefulness. The process also includes noise removal, normalization, and linguistic processing to deal with linguistic variances and background noise in the discourse. The Kernel Principal Component Analysis (K-PCA) is employed for feature extraction, aiming to derive valuable attributes from each modality and encode labels for array values. We propose a Bi-LG-GCN-based architecture explicitly tailored for multimodal emotion recognition, effectively fusing data from various modalities. The Bi-LG-GCN system takes each modality's feature-extracted and pre-processed representation as input to the generator network, generating realistic synthetic data samples that capture multimodal relationships. These generated synthetic data samples, reflecting multimodal relationships, serve as inputs to the discriminator network, which has been trained to distinguish genuine from synthetic data. With this approach, the model can learn discriminative features for emotion recognition and make accurate predictions regarding subsequent emotional states. Our method was evaluated on the MELD dataset, yielding notable results in terms of accuracy (80%), F1-score (81%), precision (81%), and recall (81%) when using the MELD dataset. The pre-processing and feature extraction steps enhance input representation quality and discrimination. Our Bi-LG-GCN-based approach, featuring multimodal data synthesis, outperforms contemporary techniques, thus demonstrating its practical utility.

Zitieren

APA

Alsaadawı, H. F. T., & Daş, R. (2024). Multimodal Emotion Recognition Using Bi-LG-GCN for MELD Dataset. Balkan Journal of Electrical and Computer Engineering, 12(1), 36-46. https://doi.org/10.17694/bajece.1372107

Vancouver

Alsaadawı HFT, Daş R. Multimodal Emotion Recognition Using Bi-LG-GCN for MELD Dataset. Balkan Journal of Electrical and Computer Engineering. 2024;12(1):36-46. doi:10.17694/bajece.1372107

BibTeX
@article{Alsaadaw2024,
  author = {Alsaadawı, Hussein Farooq Tayeb and Daş, Resul},
  title = {Multimodal Emotion Recognition Using Bi-LG-GCN for MELD Dataset},
  journal = {Balkan Journal of Electrical and Computer Engineering},
  year = {2024},
  volume = {12},
  number = {1},
  pages = {36--46},
  doi = {10.17694/bajece.1372107}
}
Kostenlos starten →