EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

GFRN-SEA: Global-Aware Feature Representation Network for Speech Emotion Analysis

Lei Pan, Qi Wang

IEEE Access · 2024 · 12 · 161500-161518

DOI: 10.1109/access.2024.3490186 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Access, 2024. Verfasst von Lei Pan und 1 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 66 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Research Foundation of Nanjing Institute of Technology.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the rapid advancement of artificial intelligence and machine learning, speech emotion recognition (SER) holds significant potential across various applications. Despite progress, challenges persist in effectively extracting and fusing deep features from original audio data. This paper presents a Global-Aware Feature Representation Network for Speech Emotion Analysis (GFRN-SEA), a novel SER model that integrates three distinct levels of audio features: Mel-Frequency Cepstral Coefficients (MFCC), spectrogram features, and HuBERT features. By processing the spectrogram through a ResNet encoder, MFCC through a BiLSTM encoder, and raw audio waveforms through the HuBERT model, we ensure a comprehensive capture of the temporal, spatial, and spatiotemporal features of the audio signal. To further refine the extracted features, GFRN-SEA introduces a Multi-dimensional and Multi-scale (MDMS) convolution module. This module applies convolutions at multiple scales, capturing fine-grained details and broader contextual information from spectrogram and MFCC features. Another key component of GFRN-SEA is the multi-layer cross attention (MLCA) mechanism. The MLCA effectively fuses extracted features by emphasizing the most relevant information across modalities and integrating spectrogram, MFCC, and HuBERT features to enhance synergy and improve emotion recognition performance. Finally, the fused features are input into a fully connected layer to classify the emotional state of the speaker. Our method leverages the complementary strengths of multi-dimension and multi-scale feature extraction and sophisticated fusion techniques, achieving state-of-the-art performance on four mainstream public datasets.

Förderung

  • Research Foundation of Nanjing Institute of Technology

Zitieren

APA

Pan, L., & Wang, Q. (2024). GFRN-SEA: Global-Aware Feature Representation Network for Speech Emotion Analysis. IEEE Access, 12, 161500-161518. https://doi.org/10.1109/access.2024.3490186

Vancouver

Pan L, Wang Q. GFRN-SEA: Global-Aware Feature Representation Network for Speech Emotion Analysis. IEEE Access. 2024;12:161500-161518. doi:10.1109/access.2024.3490186

BibTeX
@article{Pan2024,
  author = {Pan, Lei and Wang, Qi},
  title = {GFRN-SEA: Global-Aware Feature Representation Network for Speech Emotion Analysis},
  journal = {IEEE Access},
  year = {2024},
  volume = {12},
  pages = {161500--161518},
  doi = {10.1109/access.2024.3490186}
}
Kostenlos starten →