Dance2MIDI: Dance-driven multi-instrument music generation
Computational Visual Media · 2024-08 · 10 (4) · 791-802
DOI: 10.1007/s41095-024-0417-1 ↗Kurz gefasst
Fachartikel in der Zeitschrift Computational Visual Media, 2024-08. Verfasst von Bo Han und 4 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Dance-driven music generation aims to generate musical pieces conditioned on dance videos. Previous works focus on monophonic or raw audio generation, while the multi-instrument scenario is under-explored. The challenges associated with dance-driven multi-instrument music (MIDI) generation are twofold: (i) lack of a publicly available multi-instrument MIDI and video paired dataset and (ii) the weak correlation between music and video. To tackle these challenges, we have built the first multi-instrument MIDI and dance paired dataset (D2MIDI). Based on this dataset, we introduce a multi-instrument MIDI generation framework (Dance2MIDI) conditioned on dance video. Firstly, to capture the relationship between dance and music, we employ a graph convolutional network to encode the dance motion. This allows us to extract features related to dance movement and dance style. Secondly, to generate a harmonious rhythm, we utilize a transformer model to decode the drum track sequence, leveraging a cross-attention mechanism. Thirdly, we model the task of generating the remaining tracks based on the drum track as a sequence understanding and completion task. A BERT-like model is employed to comprehend the context of the entire music piece through self-supervised learning. We evaluate the music generated by our framework trained on the D2MIDI dataset and demonstrate that our method achieves state-of-the-art performance.
Zitieren
APA
Han, B., Li, Y., Shen, Y., Ren, Y., & Han, F. (2024). Dance2MIDI: Dance-driven multi-instrument music generation. Computational Visual Media, 10(4), 791-802. https://doi.org/10.1007/s41095-024-0417-1
Vancouver
Han B, Li Y, Shen Y, Ren Y, Han F. Dance2MIDI: Dance-driven multi-instrument music generation. Computational Visual Media. 2024;10(4):791-802. doi:10.1007/s41095-024-0417-1
BibTeX
@article{Han2024,
author = {Han, Bo and Li, Yuheng and Shen, Yixuan and Ren, Yi and Han, Feilin},
title = {Dance2MIDI: Dance-driven multi-instrument music generation},
journal = {Computational Visual Media},
year = {2024},
volume = {10},
number = {4},
pages = {791--802},
doi = {10.1007/s41095-024-0417-1}
}
EUVIMED