EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Generating Prosody-Aligned Gestures via Residual Vector Quantization With Uniform Regularization and Activity Loss

Hyoungki Choi, Hyeongseok Choi, Jinbeum Jang, Joonki Paik

IEEE Access · 2026 · 14 · 98227-98239

DOI: 10.1109/access.2026.3707129 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Access, 2026. Verfasst von Hyoungki Choi und 3 weiteren Autor:innen. Die Arbeit nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government, National Research Foundation of Korea (NRF) grant funded by the Korea government, MSIT (Ministry of Science and ICT), Korea, under the Graduate School of Metaverse Convergence support program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Co-speech gesture generation (CSGG) aims to synthesize gestures that reflect both the semantic content and prosodic features of spoken language. While recent data-driven methods, such as diffusion-based models, have improved the realism of generated gestures, they often face challenges such as difficulties in deterministic alignment with speech prosody, high computational costs, and inference latency. Specifically, synchronizing gestures with fine-grained prosodic cues remains a challenge due to limitations in the structure of latent representations. To address this, we propose a two-stage framework consisting of a gesture embedding network and a gesture generator. The embedding network utilizes residual vector quantization (RVQ) with multiple codebooks to encode motion segments into discrete representations that preserve both macro-dynamics and micro-details of human gestures. To ensure diverse and efficient codebook usage, we introduce a uniform regularization (UR) objective that mitigates codebook collapse. We also design an activity loss (AL) that preserves the motion energy of the original gestures, enabling the model to respond adaptively to speech rhythm and intensity through deterministic mapping. Experimental results across several benchmark datasets indicate that our method achieves competitive or superior performance compared to representative state-of-the-art (SOTA) models, including diffusion-based approaches, in terms of Fréchet Gesture Distance (FGD) and motion diversity, while offering substantial improvements in inference efficiency. These findings validate the effectiveness and efficiency of our model in generating natural, expressive, and rhythm-aware co-speech gestures.

Förderung

  • Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government
  • National Research Foundation of Korea (NRF) grant funded by the Korea government
  • MSIT (Ministry of Science and ICT), Korea, under the Graduate School of Metaverse Convergence support program

Zitieren

APA

Choi, H., Choi, H., Jang, J., & Paik, J. (2026). Generating Prosody-Aligned Gestures via Residual Vector Quantization With Uniform Regularization and Activity Loss. IEEE Access, 14, 98227-98239. https://doi.org/10.1109/access.2026.3707129

Vancouver

Choi H, Choi H, Jang J, Paik J. Generating Prosody-Aligned Gestures via Residual Vector Quantization With Uniform Regularization and Activity Loss. IEEE Access. 2026;14:98227-98239. doi:10.1109/access.2026.3707129

BibTeX
@article{Choi2026,
  author = {Choi, Hyoungki and Choi, Hyeongseok and Jang, Jinbeum and Paik, Joonki},
  title = {Generating Prosody-Aligned Gestures via Residual Vector Quantization With Uniform Regularization and Activity Loss},
  journal = {IEEE Access},
  year = {2026},
  volume = {14},
  pages = {98227--98239},
  doi = {10.1109/access.2026.3707129}
}
Kostenlos starten →