EUVIMEDEUVIMED entdecken
Crossref

G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model

Pan Xie, Qipeng Zhang, Peng Taiying, Hao Tang, Yao Du, Zexian Li

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (6) · 6234-6242

DOI: 10.1609/aaai.v38i6.28441 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Pan Xie und 5 weiteren Autor:innen. Die Arbeit wurde bisher 27-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The Sign Language Production (SLP) project aims to automatically translate spoken languages into sign sequences. Our approach focuses on the transformation of sign gloss sequences into their corresponding sign pose sequences (G2P). In this paper, we present a novel solution for this task by converting the continuous pose space generation problem into a discrete sequence generation problem. We introduce the Pose-VQVAE framework, which combines Variational Autoencoders (VAEs) with vector quantization to produce a discrete latent representation for continuous pose sequences. Additionally, we propose the G2P-DDM model, a discrete denoising diffusion architecture for length-varied discrete sequence data, to model the latent prior. To further enhance the quality of pose sequence generation in the discrete space, we present the CodeUnet model to leverage spatial-temporal information. Lastly, we develop a heuristic sequential clustering method to predict variable lengths of pose sequences for corresponding gloss sequences. Our results show that our model outperforms state-of-the-art G2P models on the public SLP evaluation benchmark. For more generated results, please visit our project page: https://slpdiffusier.github.io/g2p-ddm.

Zitieren

APA

Xie, P., Zhang, Q., Taiying, P., Tang, H., Du, Y., & Li, Z. (2024). G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6), 6234-6242. https://doi.org/10.1609/aaai.v38i6.28441

Vancouver

Xie P, Zhang Q, Taiying P, Tang H, Du Y, Li Z. G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(6):6234-6242. doi:10.1609/aaai.v38i6.28441

BibTeX
@article{Xie2024,
  author = {Xie, Pan and Zhang, Qipeng and Taiying, Peng and Tang, Hao and Du, Yao and Li, Zexian},
  title = {G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {6},
  pages = {6234--6242},
  doi = {10.1609/aaai.v38i6.28441}
}
Kostenlos starten →