MELT: Optimization-Driven Music Emotion Learning with Temporal Token-Level Fusion
Mathematics · 2026-05-15 · 14 (10) · 1690
DOI: 10.3390/math14101690 ↗At a glance
Journal article in Mathematics, published online on 2026-05-15. Written by Yihe Yin and 2 co-authors. 50 references. The full text is freely available (open access, CC BY 4.0). Funded by: Research Fund of Macao University of Science and Technology, Science and Technology Development Fund of Macau Project.
Abstract (original language)
Music emotion recognition (MER) can be formulated as a multimodal optimization problem that predicts an emotion label from coupled audio and lyric sequences. Existing methods typically perform unimodal learning or coarse global fusion, which overlooks fine-grained temporal-token correspondences between musical dynamics and lyric semantics. We propose MELT (Music Emotion Learning with Temporal token-level fusion), an optimization-driven framework with four modules: a BERT-based lyrics semantic encoder (LSE), a segment temporal encoder (STE) that models audio-segment dependencies via a Transformer, a token-level temporal fusion (TTF) module with gated cross-attention, and an emotion mood head (EMH) for four-class prediction. Training is conducted end-to-end by jointly minimizing a supervised classification term and an auxiliary cross-modal contrastive alignment term, yielding a unified objective that improves both class separability and representation consistency. On the MoodyLyrics benchmark, MELT achieves 87.6% weighted F1 for four-class emotion recognition (angry, happy, relaxed, sad), outperforming unimodal baselines and representative early/late fusion strategies. Ablation results further verify that temporal encoding, gated token-level fusion, and joint optimization each contribute to the final performance.
Funding
- Research Fund of Macao University of Science and Technology
- Science and Technology Development Fund of Macau Project
Cite
APA
Yin, Y., Tian, Z., & Chen, J. (2026). MELT: Optimization-Driven Music Emotion Learning with Temporal Token-Level Fusion. Mathematics, 14(10), 1690. https://doi.org/10.3390/math14101690
Vancouver
Yin Y, Tian Z, Chen J. MELT: Optimization-Driven Music Emotion Learning with Temporal Token-Level Fusion. Mathematics. 2026;14(10):1690. doi:10.3390/math14101690
BibTeX
@article{Yin2026,
author = {Yin, Yihe and Tian, Zhen and Chen, Junming},
title = {MELT: Optimization-Driven Music Emotion Learning with Temporal Token-Level Fusion},
journal = {Mathematics},
year = {2026},
volume = {14},
number = {10},
pages = {1690},
doi = {10.3390/math14101690}
}
EUVIMED