EUVIMEDEUVIMED entdecken
Crossref

Bailando ++: 3D Dance GPT With Choreographic Memory

Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, Ziwei Liu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2023-12 · 45 (12) · 14192-14207

DOI: 10.1109/tpami.2023.3319435 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023-12. Verfasst von Li Siyao und 7 weiteren Autor:innen. Die Arbeit wurde bisher 35-mal zitiert (Crossref) und nennt 54 Literaturangaben. Gefördert von: RIE2020 Industry Alignment Fund Industry Collaboration Projects, National Research Foundation Singapore, AI Singapore Programme.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Our proposed music-to-dance framework, Bailando++, addresses the challenges of driving 3D characters to dance in a way that follows the constraints of choreography norms and maintains temporal coherency with different music genres. Bailando++ consists of two components: a choreographic memory that learns to summarize meaningful dancing units from 3D pose sequences, and an actor-critic Generative Pre-trained Transformer (GPT) that composes these units into a fluent dance coherent to the music. In particular, to synchronize the diverse motion tempos and music beats, we introduce an actor-critic-based reinforcement learning scheme to the GPT with a novel beat-align reward function. Additionally, we consider learning human dance poses in the rotation domain to avoid body distortions incompatible with human morphology, and introduce a musical contextual encoding to allow the motion GPT to grasp longer-term patterns of music. Our experiments on the standard benchmark show that Bailando++ achieves state-of-the-art performance both qualitatively and quantitatively, with the added benefit of the unsupervised discovery of human-interpretable dancing-style poses in the choreographic memory.

Förderung

  • RIE2020 Industry Alignment Fund Industry Collaboration Projects
  • National Research Foundation Singapore
  • AI Singapore Programme
  • NTU NAP

Zitieren

APA

Siyao, L., Yu, W., Gu, T., Lin, C., Wang, Q., Qian, C., Loy, C. C., & Liu, Z. (2023). Bailando ++: 3D Dance GPT With Choreographic Memory. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(12), 14192-14207. https://doi.org/10.1109/tpami.2023.3319435

Vancouver

Siyao L, Yu W, Gu T, Lin C, Wang Q, Qian C, et al. Bailando ++: 3D Dance GPT With Choreographic Memory. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(12):14192-14207. doi:10.1109/tpami.2023.3319435

BibTeX
@article{Siyao2023,
  author = {Siyao, Li and Yu, Weijiang and Gu, Tianpei and Lin, Chunze and Wang, Quan and Qian, Chen and Loy, Chen Change and Liu, Ziwei},
  title = {Bailando ++: 3D Dance GPT With Choreographic Memory},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2023},
  volume = {45},
  number = {12},
  pages = {14192--14207},
  doi = {10.1109/tpami.2023.3319435}
}
Kostenlos starten →