Multi-Input Speech Emotion Recognition Model Using Mel Spectrogram and GeMAPS
Sensors · 03.02.2023 · 23 (3) · 1743
DOI: 10.3390/s23031743 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 03.02.2023. Verfasst von Itsuki Toyoshima und 4 weiteren Autor:innen. Beteiligte Einrichtung: Division of Information and Electronic Engineering, Muroran Institute of Technology, 27-1, Mizumoto-cho, Muroran 050-8585, Hokkaido, Japan. Hauptthemen laut MEDLINE: Speech, Emotions. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 54 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
The existing research on emotion recognition commonly uses mel spectrogram (MelSpec) and Geneva minimalistic acoustic parameter set (GeMAPS) as acoustic parameters to learn the audio features. MelSpec can represent the time-series variations of each frequency but cannot manage multiple types of audio features. On the other hand, GeMAPS can handle multiple audio features but fails to provide information on their time-series variations. Thus, this study proposes a speech emotion recognition model based on a multi-input deep neural network that simultaneously learns these two audio features. The proposed model comprises three parts, specifically, for learning MelSpec in image format, learning GeMAPS in vector format, and integrating them to predict the emotion. Additionally, a focal loss function is introduced to address the imbalanced data problem among the emotion classes. The results of the recognition experiments demonstrate weighted and unweighted accuracies of 0.6657 and 0.6149, respectively, which are higher than or comparable to those of the existing state-of-the-art methods. Overall, the proposed model significantly improves the recognition accuracy of the emotion “happiness”, which has been difficult to identify in previous studies owing to limited data. Therefore, the proposed model can effectively recognize emotions from speech and can be applied for practical purposes with future development.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Division of Information and Electronic Engineering, Muroran Institute of Technology, 27-1, Mizumoto-cho, Muroran 050-8585, Hokkaido, Japan.
- College of Information and Systems, Muroran Institute of Technology, 27-1, Mizumoto-cho, Muroran 050-8585, Hokkaido, Japan.
Zitieren
APA
Toyoshima, I., Okada, Y., Ishimaru, M., Uchiyama, R., & Tada, M. (2023). Multi-Input Speech Emotion Recognition Model Using Mel Spectrogram and GeMAPS. Sensors, 23(3), 1743. https://doi.org/10.3390/s23031743
Vancouver
Toyoshima I, Okada Y, Ishimaru M, Uchiyama R, Tada M. Multi-Input Speech Emotion Recognition Model Using Mel Spectrogram and GeMAPS. Sensors. 2023;23(3):1743. doi:10.3390/s23031743
BibTeX
@article{Toyoshima2023,
author = {Toyoshima, Itsuki and Okada, Yoshifumi and Ishimaru, Momoko and Uchiyama, Ryunosuke and Tada, Mayu},
title = {Multi-Input Speech Emotion Recognition Model Using Mel Spectrogram and GeMAPS},
journal = {Sensors},
year = {2023},
volume = {23},
number = {3},
pages = {1743},
doi = {10.3390/s23031743}
}
EUVIMED