EUVIMEDPoznaj EUVIMED
Open access · CC BY 4.0

MELT: Optimization-Driven Music Emotion Learning with Temporal Token-Level Fusion

Yihe Yin, Zhen Tian, Junming Chen

Mathematics · 2026-05-15 · 14 (10) · 1690

DOI: 10.3390/math14101690 ↗

At a glance

Journal article in Mathematics, published online on 2026-05-15. Written by Yihe Yin and 2 co-authors. 50 references. The full text is freely available (open access, CC BY 4.0). Funded by: Research Fund of Macao University of Science and Technology, Science and Technology Development Fund of Macau Project.

AbstraktDane bibliograficzneCiteRelated studies

Abstract (original language)

Music emotion recognition (MER) can be formulated as a multimodal optimization problem that predicts an emotion label from coupled audio and lyric sequences. Existing methods typically perform unimodal learning or coarse global fusion, which overlooks fine-grained temporal-token correspondences between musical dynamics and lyric semantics. We propose MELT (Music Emotion Learning with Temporal token-level fusion), an optimization-driven framework with four modules: a BERT-based lyrics semantic encoder (LSE), a segment temporal encoder (STE) that models audio-segment dependencies via a Transformer, a token-level temporal fusion (TTF) module with gated cross-attention, and an emotion mood head (EMH) for four-class prediction. Training is conducted end-to-end by jointly minimizing a supervised classification term and an auxiliary cross-modal contrastive alignment term, yielding a unified objective that improves both class separability and representation consistency. On the MoodyLyrics benchmark, MELT achieves 87.6% weighted F1 for four-class emotion recognition (angry, happy, relaxed, sad), outperforming unimodal baselines and representative early/late fusion strategies. Ablation results further verify that temporal encoding, gated token-level fusion, and joint optimization each contribute to the final performance.

Funding

  • Research Fund of Macao University of Science and Technology
  • Science and Technology Development Fund of Macau Project

Cite

APA

Yin, Y., Tian, Z., & Chen, J. (2026). MELT: Optimization-Driven Music Emotion Learning with Temporal Token-Level Fusion. Mathematics, 14(10), 1690. https://doi.org/10.3390/math14101690

Vancouver

Yin Y, Tian Z, Chen J. MELT: Optimization-Driven Music Emotion Learning with Temporal Token-Level Fusion. Mathematics. 2026;14(10):1690. doi:10.3390/math14101690

BibTeX
@article{Yin2026,
  author = {Yin, Yihe and Tian, Zhen and Chen, Junming},
  title = {MELT: Optimization-Driven Music Emotion Learning with Temporal Token-Level Fusion},
  journal = {Mathematics},
  year = {2026},
  volume = {14},
  number = {10},
  pages = {1690},
  doi = {10.3390/math14101690}
}
Start for free →