EUVIMEDEUVIMED entdecken
Crossref

GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents

Tenglong Ao, Zeyi Zhang, Libin Liu

ACM Transactions on Graphics · 2023-08 · 42 (4) · 1-18

DOI: 10.1145/3592097 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 26.07.2023. Verfasst von Tenglong Ao und 2 weiteren Autor:innen. Die Arbeit wurde bisher 167-mal zitiert (Crossref) und nennt 88 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The automatic generation of stylized co-speech gestures has recently received increasing attention. Previous systems typically allow style control via predefined text labels or example motion clips, which are often not flexible enough to convey user intent accurately. In this work, we present GestureDiffuCLIP, a neural network framework for synthesizing realistic, stylized co-speech gestures with flexible style control. We leverage the power of the large-scale Contrastive-Language-Image-Pre-training (CLIP) model and present a novel CLIP-guided mechanism that extracts efficient style representations from multiple input modalities, such as a piece of text, an example motion clip, or a video. Our system learns a latent diffusion model to generate high-quality gestures and infuses the CLIP representations of style into the generator via an adaptive instance normalization (AdaIN) layer. We further devise a gesture-transcript alignment mechanism that ensures a semantically correct gesture generation based on contrastive learning. Our system can also be extended to allow fine-grained style control of individual body parts. We demonstrate an extensive set of examples showing the flexibility and generalizability of our model to a variety of style descriptions. In a user study, we show that our system outperforms the state-of-the-art approaches regarding human likeness, appropriateness, and style correctness.

Zitieren

APA

Ao, T., Zhang, Z., & Liu, L. (2023). GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents. ACM Transactions on Graphics, 42(4), 1-18. https://doi.org/10.1145/3592097

Vancouver

Ao T, Zhang Z, Liu L. GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents. ACM Transactions on Graphics. 2023;42(4):1-18. doi:10.1145/3592097

BibTeX
@article{Ao2023,
  author = {Ao, Tenglong and Zhang, Zeyi and Liu, Libin},
  title = {GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents},
  journal = {ACM Transactions on Graphics},
  year = {2023},
  volume = {42},
  number = {4},
  pages = {1--18},
  doi = {10.1145/3592097}
}
Kostenlos starten →