EUVIMEDEUVIMED entdecken
Crossref

DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation

Jisoo Kim, Jungbin Cho, Joonho Park, Soonmin Hwang, Da Eun Kim, Geon Kim, Youngjae Yu

Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (4) · 4275-4283

DOI: 10.1609/aaai.v39i4.32449 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Jisoo Kim und 6 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Speech-driven 3D facial animation has garnered lots of attention thanks to its broad range of applications. Despite recent advancements in achieving realistic lip motion, current methods fail to capture the nuanced emotional undertones conveyed through speech and produce monotonous facial motion. These limitations result in blunt and repetitive facial animations, reducing user engagement and hindering their applicability. To address these challenges, we introduce DEEPTalk, a novel approach that generates diverse and emotionally rich 3D facial expressions directly from speech inputs. To achieve this, we first train DEE (Dynamic Emotion Embedding), which employs probabilistic contrastive learning to forge a joint emotion embedding space for both speech and facial motion. This probabilistic framework captures the uncertainty in interpreting emotions from speech and facial motion, enabling the derivation of emotion vectors from its multifaceted space. Moreover, to generate dynamic facial motion, we design TH-VQVAE (Temporally Hierarchical VQ-VAE) as an expressive and robust motion prior overcoming limitations of VAEs and VQ-VAEs. Utilizing these strong priors, we develop DEEPTalk, a talking head generator that non-autoregressively predicts codebook indices to create dynamic facial motion, incorporating a novel emotion consistency loss. Extensive experiments on various datasets demonstrate the effectiveness of our approach in creating diverse, emotionally expressive talking faces that maintain accurate lip-sync. Our project page is available at https://whwjdqls.github.io/deeptalk.github.io/.

Zitieren

APA

Kim, J., Cho, J., Park, J., Hwang, S., Kim, D. E., Kim, G., & Yu, Y. (2025). DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation. Proceedings of the AAAI Conference on Artificial Intelligence, 39(4), 4275-4283. https://doi.org/10.1609/aaai.v39i4.32449

Vancouver

Kim J, Cho J, Park J, Hwang S, Kim DE, Kim G, et al. DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(4):4275-4283. doi:10.1609/aaai.v39i4.32449

BibTeX
@article{Kim2025,
  author = {Kim, Jisoo and Cho, Jungbin and Park, Joonho and Hwang, Soonmin and Kim, Da Eun and Kim, Geon and Yu, Youngjae},
  title = {DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2025},
  volume = {39},
  number = {4},
  pages = {4275--4283},
  doi = {10.1609/aaai.v39i4.32449}
}
Kostenlos starten →