EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Dance2MIDI: Dance-driven multi-instrument music generation

Bo Han, Yuheng Li, Yixuan Shen, Yi Ren, Feilin Han

Computational Visual Media · 2024-08 · 10 (4) · 791-802

DOI: 10.1007/s41095-024-0417-1 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computational Visual Media, 2024-08. Verfasst von Bo Han und 4 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Dance-driven music generation aims to generate musical pieces conditioned on dance videos. Previous works focus on monophonic or raw audio generation, while the multi-instrument scenario is under-explored. The challenges associated with dance-driven multi-instrument music (MIDI) generation are twofold: (i) lack of a publicly available multi-instrument MIDI and video paired dataset and (ii) the weak correlation between music and video. To tackle these challenges, we have built the first multi-instrument MIDI and dance paired dataset (D2MIDI). Based on this dataset, we introduce a multi-instrument MIDI generation framework (Dance2MIDI) conditioned on dance video. Firstly, to capture the relationship between dance and music, we employ a graph convolutional network to encode the dance motion. This allows us to extract features related to dance movement and dance style. Secondly, to generate a harmonious rhythm, we utilize a transformer model to decode the drum track sequence, leveraging a cross-attention mechanism. Thirdly, we model the task of generating the remaining tracks based on the drum track as a sequence understanding and completion task. A BERT-like model is employed to comprehend the context of the entire music piece through self-supervised learning. We evaluate the music generated by our framework trained on the D2MIDI dataset and demonstrate that our method achieves state-of-the-art performance.

Zitieren

APA

Han, B., Li, Y., Shen, Y., Ren, Y., & Han, F. (2024). Dance2MIDI: Dance-driven multi-instrument music generation. Computational Visual Media, 10(4), 791-802. https://doi.org/10.1007/s41095-024-0417-1

Vancouver

Han B, Li Y, Shen Y, Ren Y, Han F. Dance2MIDI: Dance-driven multi-instrument music generation. Computational Visual Media. 2024;10(4):791-802. doi:10.1007/s41095-024-0417-1

BibTeX
@article{Han2024,
  author = {Han, Bo and Li, Yuheng and Shen, Yixuan and Ren, Yi and Han, Feilin},
  title = {Dance2MIDI: Dance-driven multi-instrument music generation},
  journal = {Computational Visual Media},
  year = {2024},
  volume = {10},
  number = {4},
  pages = {791--802},
  doi = {10.1007/s41095-024-0417-1}
}
Kostenlos starten →