EUVIMEDEUVIMED entdecken
Crossref

UniMuMo: Unified Text, Music, and Motion Generation

Han Yang, Kun Su, Yutong Zhang, Jiaben Chen, Kaizhi Qian, Gaowen Liu, Chuang Gan

Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (24) · 25615-25623

DOI: 10.1609/aaai.v39i24.34752 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Han Yang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We introduce UniMuMo, a unified multimodal model capable of taking arbitrary text, music, and motion data as input conditions to generate outputs across all three modalities. To address the lack of time-synchronized data, we align unpaired music and motion data based on rhythmic patterns to leverage existing large-scale music-only and motion-only datasets. By converting music, motion, and text into token-based representation, our model bridges these modalities through a unified encoder-decoder transformer architecture. To support multiple generation tasks within a single framework, we introduce several architectural improvements. We propose encoding motion with a music codebook, mapping motion into the same feature space as music. We introduce a music-motion parallel generation scheme that unifies all music and motion generation tasks into a single transformer decoder architecture with a single training task of music-motion joint generation. Moreover, the model is designed by fine-tuning existing pre-trained single-modality models, significantly reducing computational demands. Extensive experiments demonstrate that UniMuMo achieves competitive results on all unidirectional generation benchmarks across music, motion, and text modalities.

Zitieren

APA

Yang, H., Su, K., Zhang, Y., Chen, J., Qian, K., Liu, G., & Gan, C. (2025). UniMuMo: Unified Text, Music, and Motion Generation. Proceedings of the AAAI Conference on Artificial Intelligence, 39(24), 25615-25623. https://doi.org/10.1609/aaai.v39i24.34752

Vancouver

Yang H, Su K, Zhang Y, Chen J, Qian K, Liu G, et al. UniMuMo: Unified Text, Music, and Motion Generation. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(24):25615-25623. doi:10.1609/aaai.v39i24.34752

BibTeX
@article{Yang2025,
  author = {Yang, Han and Su, Kun and Zhang, Yutong and Chen, Jiaben and Qian, Kaizhi and Liu, Gaowen and Gan, Chuang},
  title = {UniMuMo: Unified Text, Music, and Motion Generation},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2025},
  volume = {39},
  number = {24},
  pages = {25615--25623},
  doi = {10.1609/aaai.v39i24.34752}
}
Kostenlos starten →