EUVIMEDEUVIMED entdecken
Crossref

Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap

Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, Bjorn W Schuller

IEEE Transactions on Pattern Analysis and Machine Intelligence · 01.09.2023 · 45 (9) · 10745-10759

DOI: 10.1109/tpami.2023.3263585 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, erschienen am 01.09.2023. Verfasst von Johannes Wagner und 6 weiteren Autor:innen. Hauptthemen laut MEDLINE: Speech, Algorithms. Die Arbeit wurde bisher 372-mal zitiert (Crossref) und nennt 67 Literaturangaben. Gefördert von: DFG's Reinhart Koselleck.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent advances in transformer-based architectures have shown promise in several machine learning tasks. In the audio domain, such architectures have been successfully utilised in the field of speech emotion recognition (SER). However, existing works have not evaluated the influence of model size and pre-training data on downstream performance, and have shown limited attention to generalisation, robustness, fairness, and efficiency. The present contribution conducts a thorough analysis of these aspects on several pre-trained variants of wav2vec 2.0 and HuBERT that we fine-tuned on the dimensions arousal, dominance, and valence of MSP-Podcast, while additionally using IEMOCAP and MOSI to test cross-corpus generalisation. To the best of our knowledge, we obtain the top performance for valence prediction without use of explicit linguistic information, with a concordance correlation coefficient (CCC) of. 638 on MSP-Podcast. Our investigations reveal that transformer-based architectures are more robust compared to a CNN-based baseline and fair with respect to gender groups, but not towards individual speakers. Finally, we show that their success on valence is based on implicit linguistic information, which explains why they perform on-par with recent multimodal approaches that explicitly utilise textual information. To make our findings reproducible, we release the best performing model to the community.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

SpeechAlgorithmsEmotionsMachine Learning

Förderung

  • DFG's Reinhart Koselleck

Zitieren

APA

Wagner, J., Triantafyllopoulos, A., Wierstorf, H., Schmitt, M., Burkhardt, F., Eyben, F., & Schuller, B. W. (2023). Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9), 10745-10759. https://doi.org/10.1109/tpami.2023.3263585

Vancouver

Wagner J, Triantafyllopoulos A, Wierstorf H, Schmitt M, Burkhardt F, Eyben F, et al. Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(9):10745-10759. doi:10.1109/tpami.2023.3263585

BibTeX
@article{Wagner2023,
  author = {Wagner, Johannes and Triantafyllopoulos, Andreas and Wierstorf, Hagen and Schmitt, Maximilian and Burkhardt, Felix and Eyben, Florian and Schuller, Bjorn W},
  title = {Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2023},
  volume = {45},
  number = {9},
  pages = {10745--10759},
  doi = {10.1109/tpami.2023.3263585}
}
Kostenlos starten →