Speech Emotion Recognition Model Based on Joint Modeling of Discrete and Dimensional Emotion Representation
Applied Sciences · 10.01.2025 · 15 (2) · 623
DOI: 10.3390/app15020623 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 10.01.2025. Verfasst von John Lorenzo Bautista und 1 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Industrial Technology Innovation Program, Ministry of Trade, Industry, and Energy (MOTIE, Korea).
Abstract (Originalsprache)
This paper introduces a novel joint model architecture for Speech Emotion Recognition (SER) that integrates both discrete and dimensional emotional representations, allowing for the simultaneous training of classification and regression tasks to improve the comprehensiveness and interpretability of emotion recognition. By employing a joint loss function that combines categorical and regression losses, the model ensures balanced optimization across tasks, with experiments exploring various weighting schemes using a tunable parameter to adjust task importance. Two adaptive weight balancing schemes, Dynamic Weighting and Joint Weighting, further enhance performance by dynamically adjusting task weights based on optimization progress and ensuring balanced emotion representation during backpropagation. The architecture employs parallel feature extraction through independent encoders, designed to capture unique features from multiple modalities, including Mel-frequency Cepstral Coefficients (MFCC), Short-term Features (STF), Mel-spectrograms, and raw audio signals. Additionally, pre-trained models such as Wav2Vec 2.0 and HuBERT are integrated to leverage their robust latent features. The inclusion of self-attention and co-attention mechanisms allows the model to capture relationships between input modalities and interdependencies among features, further improving its interpretability and integration capabilities. Experiments conducted on the IEMOCAP dataset using a leave-one-subject-out approach demonstrate the model’s effectiveness, with results showing a 1–2% accuracy improvement over classification-only models. The optimal configuration, incorporating the joint architecture, dynamic weighting, and parallel processing of multimodal features, achieves a weighted accuracy of 72.66%, an unweighted accuracy of 73.22%, and a mean Concordance Correlation Coefficient (CCC) of 0.3717. These results validate the effectiveness of the proposed joint model architecture and adaptive balancing weight schemes in improving SER performance.
Förderung
- Industrial Technology Innovation Program
- Ministry of Trade, Industry, and Energy (MOTIE, Korea)
Zitieren
APA
Bautista, J. L., & Shin, H. S. (2025). Speech Emotion Recognition Model Based on Joint Modeling of Discrete and Dimensional Emotion Representation. Applied Sciences, 15(2), 623. https://doi.org/10.3390/app15020623
Vancouver
Bautista JL, Shin HS. Speech Emotion Recognition Model Based on Joint Modeling of Discrete and Dimensional Emotion Representation. Applied Sciences. 2025;15(2):623. doi:10.3390/app15020623
BibTeX
@article{Bautista2025,
author = {Bautista, John Lorenzo and Shin, Hyun Soon},
title = {Speech Emotion Recognition Model Based on Joint Modeling of Discrete and Dimensional Emotion Representation},
journal = {Applied Sciences},
year = {2025},
volume = {15},
number = {2},
pages = {623},
doi = {10.3390/app15020623}
}
EUVIMED