GFRN-SEA: Global-Aware Feature Representation Network for Speech Emotion Analysis
IEEE Access · 2024 · 12 · 161500-161518
DOI: 10.1109/access.2024.3490186 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Access, 2024. Verfasst von Lei Pan und 1 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 66 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Research Foundation of Nanjing Institute of Technology.
Abstract (Originalsprache)
With the rapid advancement of artificial intelligence and machine learning, speech emotion recognition (SER) holds significant potential across various applications. Despite progress, challenges persist in effectively extracting and fusing deep features from original audio data. This paper presents a Global-Aware Feature Representation Network for Speech Emotion Analysis (GFRN-SEA), a novel SER model that integrates three distinct levels of audio features: Mel-Frequency Cepstral Coefficients (MFCC), spectrogram features, and HuBERT features. By processing the spectrogram through a ResNet encoder, MFCC through a BiLSTM encoder, and raw audio waveforms through the HuBERT model, we ensure a comprehensive capture of the temporal, spatial, and spatiotemporal features of the audio signal. To further refine the extracted features, GFRN-SEA introduces a Multi-dimensional and Multi-scale (MDMS) convolution module. This module applies convolutions at multiple scales, capturing fine-grained details and broader contextual information from spectrogram and MFCC features. Another key component of GFRN-SEA is the multi-layer cross attention (MLCA) mechanism. The MLCA effectively fuses extracted features by emphasizing the most relevant information across modalities and integrating spectrogram, MFCC, and HuBERT features to enhance synergy and improve emotion recognition performance. Finally, the fused features are input into a fully connected layer to classify the emotional state of the speaker. Our method leverages the complementary strengths of multi-dimension and multi-scale feature extraction and sophisticated fusion techniques, achieving state-of-the-art performance on four mainstream public datasets.
Förderung
- Research Foundation of Nanjing Institute of Technology
Zitieren
APA
Pan, L., & Wang, Q. (2024). GFRN-SEA: Global-Aware Feature Representation Network for Speech Emotion Analysis. IEEE Access, 12, 161500-161518. https://doi.org/10.1109/access.2024.3490186
Vancouver
Pan L, Wang Q. GFRN-SEA: Global-Aware Feature Representation Network for Speech Emotion Analysis. IEEE Access. 2024;12:161500-161518. doi:10.1109/access.2024.3490186
BibTeX
@article{Pan2024,
author = {Pan, Lei and Wang, Qi},
title = {GFRN-SEA: Global-Aware Feature Representation Network for Speech Emotion Analysis},
journal = {IEEE Access},
year = {2024},
volume = {12},
pages = {161500--161518},
doi = {10.1109/access.2024.3490186}
}
EUVIMED