EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Multimodal Emotion Recognition Using Feature Fusion: An LLM-Based Approach

Omkumar Chandraumakantham, N. Gowtham, Mohammed Zakariah, Abdulaziz Almazyad

IEEE Access · 2024 · 12 · 108052-108071

DOI: 10.1109/access.2024.3425953 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Access, 2024. Verfasst von Omkumar Chandraumakantham und 3 weiteren Autor:innen. Die Arbeit wurde bisher 45-mal zitiert (Crossref) und nennt 28 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Researchers Supporting Program at King Saud University.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal emotion recognition is a developing field that analyzes emotions through various channels, mainly audio, video, and text. However, existing state-of-the-art systems focus on two to three modalities at the most, utilize traditional techniques, fail to consider emotional interplay, lack the scope to add more modalities, and aren’t efficient in predicting emotions accurately. This research proposes a novel approach using rule-based systems to convert non-verbal cues to text, inspired by a limited prior attempt that lacked proper benchmarking. It achieves efficient multimodal emotion recognition by utilizing distilRoBERTa, a large language model fine-tuned with a combined textual representation of audio (such as loudness, spectral flux, MFCCs, pitch stability, and emphasis) and visual features (action units) extracted from videos. This approach is evaluated using the datasets RAVDESS and BAUM-1. It achieves high accuracy (93.18% in RAVDESS and 93.69% in BAUM-1) on both datasets, performing on par with the SOTA (state-of-the-art) systems, if not slightly better. Furthermore, the research highlights the potential for incorporating additional modalities by transforming them into text using rule-based systems and utilizing them to refine further pre-trained large language models, giving rise to a more comprehensive approach to emotion recognition.

Förderung

  • Researchers Supporting Program at King Saud University

Zitieren

APA

Chandraumakantham, O., Gowtham, N., Zakariah, M., & Almazyad, A. (2024). Multimodal Emotion Recognition Using Feature Fusion: An LLM-Based Approach. IEEE Access, 12, 108052-108071. https://doi.org/10.1109/access.2024.3425953

Vancouver

Chandraumakantham O, Gowtham N, Zakariah M, Almazyad A. Multimodal Emotion Recognition Using Feature Fusion: An LLM-Based Approach. IEEE Access. 2024;12:108052-108071. doi:10.1109/access.2024.3425953

BibTeX
@article{Chandraumakantham2024,
  author = {Chandraumakantham, Omkumar and Gowtham, N. and Zakariah, Mohammed and Almazyad, Abdulaziz},
  title = {Multimodal Emotion Recognition Using Feature Fusion: An LLM-Based Approach},
  journal = {IEEE Access},
  year = {2024},
  volume = {12},
  pages = {108052--108071},
  doi = {10.1109/access.2024.3425953}
}
Kostenlos starten →