An autoencoder-based feature level fusion for speech emotion recognition
Digital Communications and Networks · 2024-10 · 10 (5) · 1341-1351
DOI: 10.1016/j.dcan.2022.10.018 ↗Kurz gefasst
Fachartikel in der Zeitschrift Digital Communications and Networks, 2024-10. Verfasst von Peng Shixin und 3 weiteren Autor:innen. Die Arbeit wurde bisher 27-mal zitiert (Crossref) und nennt 70 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Natural Science Foundation of China, Ministry of Education of the People's Republic of China, China Postdoctoral Science Foundation.
Abstract (Originalsprache)
Although speech emotion recognition is challenging, it has broad application prospects in human-computer interaction. Building a system that can accurately and stably recognize emotions from human languages can provide a better user experience. However, the current unimodal emotion feature representations are not distinctive enough to accomplish the recognition, and they do not effectively simulate the inter-modality dynamics in speech emotion recognition tasks. This paper proposes a multimodal method that utilizes both audio and semantic content for speech emotion recognition. The proposed method consists of three parts: two high-level feature extractors for text and audio modalities, and an autoencoder-based feature fusion. For audio modality, we propose a structure called Temporal Global Feature Extractor (TGFE) to extract the high-level features of the time-frequency domain relationship from the original speech signal. Considering that text lacks frequency information, we use only a Bidirectional Long Short-Term Memory network (BLSTM) and attention mechanism to simulate an intra-modal dynamic. Once these steps have been accomplished, the high-level text and audio features are sent to the autoencoder in parallel to learn their shared representation for final emotion classification. We conducted extensive experiments on three public benchmark datasets to evaluate our method. The results on Interactive Emotional Motion Capture (IEMOCAP) and Multimodal EmotionLines Dataset (MELD) outperform the existing method. Additionally, the results of CMU Multi-modal Opinion-level Sentiment Intensity (CMU-MOSI) are competitive. Furthermore, experimental results show that compared to unimodal information and autoencoder-based feature level fusion, the joint multimodal information (audio and text) improves the overall performance and can achieve greater accuracy than simple feature concatenation.
Förderung
- National Natural Science Foundation of China
- Ministry of Education of the People's Republic of China
- China Postdoctoral Science Foundation
- Natural Science Foundation of Hubei Province
- Hubei Technological Innovation Special Fund
Zitieren
APA
Shixin, P., Kai, C., Tian, T., & Jingying, C. (2024). An autoencoder-based feature level fusion for speech emotion recognition. Digital Communications and Networks, 10(5), 1341-1351. https://doi.org/10.1016/j.dcan.2022.10.018
Vancouver
Shixin P, Kai C, Tian T, Jingying C. An autoencoder-based feature level fusion for speech emotion recognition. Digital Communications and Networks. 2024;10(5):1341-1351. doi:10.1016/j.dcan.2022.10.018
BibTeX
@article{Shixin2024,
author = {Shixin, Peng and Kai, Chen and Tian, Tian and Jingying, Chen},
title = {An autoencoder-based feature level fusion for speech emotion recognition},
journal = {Digital Communications and Networks},
year = {2024},
volume = {10},
number = {5},
pages = {1341--1351},
doi = {10.1016/j.dcan.2022.10.018}
}
EUVIMED