EUVIMEDObjevte EUVIMED
Open access · CC BY 4.0

Speech Emotion Recognition Using Dual-Stream Representation and Cross-Attention Fusion

Shaode Yu, Jiajian Meng, Wenqing Fan, Ye Chen, Bing Zhu, Hang Yu, Yaoqin Xie, Qiurui Sun

Electronics · 2024-06-04 · 13 (11) · 2191

DOI: 10.3390/electronics13112191 ↗

At a glance

Journal article in Electronics, published online on 2024-06-04. Written by Shaode Yu and 7 co-authors. Cited 18 times so far (Crossref); 74 references. The full text is freely available (open access, CC BY 4.0). Funded by: National Key Research and Develop Program of China, National Natural Science Foundation of China, China-Central Eastern European Countries High Education Joint Education Project.

AbstraktBibliografické údajeCiteRelated studies

Abstract (original language)

Speech emotion recognition (SER) aims to recognize human emotions through in-depth analysis of audio signals. However, it remains challenging to encode emotional cues and to fuse the encoded cues effectively. In this study, dual-stream representation is developed, and both full training and fine-tuning of different deep networks are employed for encoding emotion patterns. Specifically, a cross-attention fusion (CAF) module is designed to integrate the dual-stream output for emotion recognition. Using different dual-stream encoders (fully training a text processing network and fine-tuning a pre-trained large language network), the CAF module is compared to other three fusion modules on three databases. The SER performance is quantified with weighted accuracy (WA), unweighted accuracy (UA), and F1-score (F1S). The experimental results suggest that the CAF outperforms the other three modules and leads to promising performance on the databases (EmoDB: WA, 97.20%; UA, 97.21%; F1S, 0.8804; IEMOCAP: WA, 69.65%; UA, 70.88%; F1S, 0.7084; RAVDESS: WA, 81.86%; UA, 82.75.21%; F1S, 0.8284). It is also found that fine-tuning a pre-trained large language network achieves superior representation than fully training a text processing network. In a future study, improved SER performance could be achieved through the development of a multi-stream representation of emotional cues and the incorporation of a multi-branch fusion mechanism for emotion recognition.

Funding

  • National Key Research and Develop Program of China
  • National Natural Science Foundation of China
  • China-Central Eastern European Countries High Education Joint Education Project
  • Shenzhen Science and Technology Program
  • Medium- and Long-term Technology Plan for Radio, Television and Online Audiovisual

Cite

APA

Yu, S., Meng, J., Fan, W., Chen, Y., Zhu, B., Yu, H., Xie, Y., & Sun, Q. (2024). Speech Emotion Recognition Using Dual-Stream Representation and Cross-Attention Fusion. Electronics, 13(11), 2191. https://doi.org/10.3390/electronics13112191

Vancouver

Yu S, Meng J, Fan W, Chen Y, Zhu B, Yu H, et al. Speech Emotion Recognition Using Dual-Stream Representation and Cross-Attention Fusion. Electronics. 2024;13(11):2191. doi:10.3390/electronics13112191

BibTeX
@article{Yu2024,
  author = {Yu, Shaode and Meng, Jiajian and Fan, Wenqing and Chen, Ye and Zhu, Bing and Yu, Hang and Xie, Yaoqin and Sun, Qiurui},
  title = {Speech Emotion Recognition Using Dual-Stream Representation and Cross-Attention Fusion},
  journal = {Electronics},
  year = {2024},
  volume = {13},
  number = {11},
  pages = {2191},
  doi = {10.3390/electronics13112191}
}
Start for free →