EUVIMEDEUVIMED entdecken
Crossref

Bodyformer: Semantics-guided 3D Body Gesture Synthesis with Transformer

Kunkun Pang, Dafei Qin, Yingruo Fan, Julian Habekost, Takaaki Shiratori, Junichi Yamagishi, Taku Komura

ACM Transactions on Graphics · 2023-08 · 42 (4) · 1-12

DOI: 10.1145/3592456 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 26.07.2023. Verfasst von Kunkun Pang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 21-mal zitiert (Crossref) und nennt 51 Literaturangaben. Gefördert von: Innovation and Technology Commission, Research Grant Council.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Automatic gesture synthesis from speech is a topic that has attracted researchers for applications in remote communication, video games and Metaverse. Learning the mapping between speech and 3D full-body gestures is difficult due to the stochastic nature of the problem and the lack of a rich cross-modal dataset that is needed for training. In this paper, we propose a novel transformer-based framework for automatic 3D body gesture synthesis from speech. To learn the stochastic nature of the body gesture during speech, we propose a variational transformer to effectively model a probabilistic distribution over gestures, which can produce diverse gestures during inference. Furthermore, we introduce a mode positional embedding layer to capture the different motion speeds in different speaking modes. To cope with the scarcity of data, we design an intra-modal pre-training scheme that can learn the complex mapping between the speech and the 3D gesture from a limited amount of data. Our system is trained with either the Trinity speech-gesture dataset or the Talking With Hands 16.2M dataset. The results show that our system can produce more realistic, appropriate, and diverse body gestures compared to existing state-of-the-art approaches.

Förderung

  • Innovation and Technology Commission
  • Research Grant Council

Zitieren

APA

Pang, K., Qin, D., Fan, Y., Habekost, J., Shiratori, T., Yamagishi, J., & Komura, T. (2023). Bodyformer: Semantics-guided 3D Body Gesture Synthesis with Transformer. ACM Transactions on Graphics, 42(4), 1-12. https://doi.org/10.1145/3592456

Vancouver

Pang K, Qin D, Fan Y, Habekost J, Shiratori T, Yamagishi J, et al. Bodyformer: Semantics-guided 3D Body Gesture Synthesis with Transformer. ACM Transactions on Graphics. 2023;42(4):1-12. doi:10.1145/3592456

BibTeX
@article{Pang2023,
  author = {Pang, Kunkun and Qin, Dafei and Fan, Yingruo and Habekost, Julian and Shiratori, Takaaki and Yamagishi, Junichi and Komura, Taku},
  title = {Bodyformer: Semantics-guided 3D Body Gesture Synthesis with Transformer},
  journal = {ACM Transactions on Graphics},
  year = {2023},
  volume = {42},
  number = {4},
  pages = {1--12},
  doi = {10.1145/3592456}
}
Kostenlos starten →