Bodyformer: Semantics-guided 3D Body Gesture Synthesis with Transformer
ACM Transactions on Graphics · 2023-08 · 42 (4) · 1-12
DOI: 10.1145/3592456 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 26.07.2023. Verfasst von Kunkun Pang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 21-mal zitiert (Crossref) und nennt 51 Literaturangaben. Gefördert von: Innovation and Technology Commission, Research Grant Council.
Abstract (Originalsprache)
Automatic gesture synthesis from speech is a topic that has attracted researchers for applications in remote communication, video games and Metaverse. Learning the mapping between speech and 3D full-body gestures is difficult due to the stochastic nature of the problem and the lack of a rich cross-modal dataset that is needed for training. In this paper, we propose a novel transformer-based framework for automatic 3D body gesture synthesis from speech. To learn the stochastic nature of the body gesture during speech, we propose a variational transformer to effectively model a probabilistic distribution over gestures, which can produce diverse gestures during inference. Furthermore, we introduce a mode positional embedding layer to capture the different motion speeds in different speaking modes. To cope with the scarcity of data, we design an intra-modal pre-training scheme that can learn the complex mapping between the speech and the 3D gesture from a limited amount of data. Our system is trained with either the Trinity speech-gesture dataset or the Talking With Hands 16.2M dataset. The results show that our system can produce more realistic, appropriate, and diverse body gestures compared to existing state-of-the-art approaches.
Förderung
- Innovation and Technology Commission
- Research Grant Council
Zitieren
APA
Pang, K., Qin, D., Fan, Y., Habekost, J., Shiratori, T., Yamagishi, J., & Komura, T. (2023). Bodyformer: Semantics-guided 3D Body Gesture Synthesis with Transformer. ACM Transactions on Graphics, 42(4), 1-12. https://doi.org/10.1145/3592456
Vancouver
Pang K, Qin D, Fan Y, Habekost J, Shiratori T, Yamagishi J, et al. Bodyformer: Semantics-guided 3D Body Gesture Synthesis with Transformer. ACM Transactions on Graphics. 2023;42(4):1-12. doi:10.1145/3592456
BibTeX
@article{Pang2023,
author = {Pang, Kunkun and Qin, Dafei and Fan, Yingruo and Habekost, Julian and Shiratori, Takaaki and Yamagishi, Junichi and Komura, Taku},
title = {Bodyformer: Semantics-guided 3D Body Gesture Synthesis with Transformer},
journal = {ACM Transactions on Graphics},
year = {2023},
volume = {42},
number = {4},
pages = {1--12},
doi = {10.1145/3592456}
}
EUVIMED