EUVIMEDEUVIMED entdecken
Open Access · CC BY-SA 4.0

Multimodal music emotion recognition in Indonesian songs based on CNN-LSTM, XLNet transformers

Andrew Steven Sams, Amalia Zahra

Bulletin of Electrical Engineering and Informatics · 01.02.2023 · 12 (1) · 355-364

DOI: 10.11591/eei.v12i1.4231 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Bulletin of Electrical Engineering and Informatics, online erschienen am 01.02.2023. Verfasst von Andrew Steven Sams und 1 weiteren Autor:innen. Die Arbeit wurde bisher 46-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-SA 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Music carries emotional information and allows the listener to feel the emotions contained in the music. This study proposes a multimodal music emotion recognition (MER) system using Indonesian song and lyrics data. In the proposed multimodal system, the audio data will use the mel spectrogram feature, and the lyrics feature will be extracted by going through the tokenizing process from XLNet. Convolutional long short term memory network (CNN-LSTM) performs the audio classification task, while XLNet transformers performs the lyrics classification task. The outputs of the two classification tasks are probability weight and actual prediction with the value of positive, neutral, and negative emotions, which are then combined using the stacking ensemble method. The combined output will be trained into an artificial neural network (ANN) model to get the best probability weight output. The multimodal system achieves the best performance with an accuracy of 80.56%. The results showed that the multimodal method of recognizing musical emotions gave better performance than the single modal method. In addition, hyperparameter tuning can affect the performance of multimodal systems.

Zitieren

APA

Sams, A. S., & Zahra, A. (2023). Multimodal music emotion recognition in Indonesian songs based on CNN-LSTM, XLNet transformers. Bulletin of Electrical Engineering and Informatics, 12(1), 355-364. https://doi.org/10.11591/eei.v12i1.4231

Vancouver

Sams AS, Zahra A. Multimodal music emotion recognition in Indonesian songs based on CNN-LSTM, XLNet transformers. Bulletin of Electrical Engineering and Informatics. 2023;12(1):355-364. doi:10.11591/eei.v12i1.4231

BibTeX
@article{Sams2023,
  author = {Sams, Andrew Steven and Zahra, Amalia},
  title = {Multimodal music emotion recognition in Indonesian songs based on CNN-LSTM, XLNet transformers},
  journal = {Bulletin of Electrical Engineering and Informatics},
  year = {2023},
  volume = {12},
  number = {1},
  pages = {355--364},
  doi = {10.11591/eei.v12i1.4231}
}
Kostenlos starten →