EUVIMEDEUVIMED entdecken
Crossref

Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based Spatiotemporal Diffusion for Audio-driven Talking Portrait

Chaolong Yang, Kai Yao, Yuyao Yan, Chenru Jiang, Weiguang Zhao, Jie Sun, Guangliang Cheng, Yifei Zhang, Bin Dong, Kaizhu Huang

International Journal of Computer Vision · 2026-03 · 134 (3)

DOI: 10.1007/s11263-025-02695-x ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 06.02.2026. Verfasst von Chaolong Yang und 9 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 48 Literaturangaben. Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Audio-driven single-image talking portrait generation plays a crucial role in virtual reality, digital human creation, and filmmaking. Existing approaches are generally categorized into keypoint-based and image-based methods. Keypoint-based methods effectively preserve character identity but struggle to capture fine facial details due to the fixed points limitation of the 3D Morphable Model. Moreover, traditional generative networks face challenges in establishing causality between audio and keypoints on limited datasets, resulting in low pose diversity. In contrast, image-based approaches produce high-quality portraits with diverse details using the diffusion network but incur identity distortion and expensive computational costs. In this work, we propose KDTalker, the first framework to combine unsupervised implicit 3D keypoint with a spatiotemporal diffusion model. Leveraging unsupervised implicit 3D keypoints, KDTalker adapts facial information densities, allowing the diffusion process to model diverse head poses and capture fine facial details flexibly. The custom-designed spatiotemporal attention mechanism ensures accurate lip synchronization, producing temporally consistent, high-quality animations while enhancing computational efficiency. Experimental results demonstrate that KDTalker achieves state-of-the-art performance regarding lip synchronization accuracy, head pose diversity, and execution efficiency.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Yang, C., Yao, K., Yan, Y., Jiang, C., Zhao, W., Sun, J., Cheng, G., Zhang, Y., Dong, B., & Huang, K. (2026). Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based Spatiotemporal Diffusion for Audio-driven Talking Portrait. International Journal of Computer Vision, 134(3). https://doi.org/10.1007/s11263-025-02695-x

Vancouver

Yang C, Yao K, Yan Y, Jiang C, Zhao W, Sun J, et al. Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based Spatiotemporal Diffusion for Audio-driven Talking Portrait. International Journal of Computer Vision. 2026;134(3). doi:10.1007/s11263-025-02695-x

BibTeX
@article{Yang2026,
  author = {Yang, Chaolong and Yao, Kai and Yan, Yuyao and Jiang, Chenru and Zhao, Weiguang and Sun, Jie and Cheng, Guangliang and Zhang, Yifei and Dong, Bin and Huang, Kaizhu},
  title = {Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based Spatiotemporal Diffusion for Audio-driven Talking Portrait},
  journal = {International Journal of Computer Vision},
  year = {2026},
  volume = {134},
  number = {3},
  doi = {10.1007/s11263-025-02695-x}
}
Kostenlos starten →