Speech-Driven 3D Facial Animation with Natural Head Movements
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.07.2026 · 22 (7) · 1-22
DOI: 10.1145/3811818 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 07.07.2026. Verfasst von Kuiyuan Sun und 4 weiteren Autor:innen. Die Arbeit nennt 67 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key R&D Program of China, National NSF of China, Fundamental Research Funds for the Central Universities.
Abstract (Originalsprache)
Speech-driven 3D facial animation has been studied for a long time, yet many challenges still prevent achieving truly natural results. One major issue is that current methods often overlook head motion during speech. To address this, we conducted a preliminary investigation and identified several key obstacles: the scarcity of 3D facial animation datasets with head motion and the limitations of regular sequence prediction models (e.g., Transformer, LSTM, and GRU), which are designed for discrete dynamic sequences and do not align well with continuous 3D head motion sequences. To solve these issues, we propose a head motion prediction module. This module uses audio and the initial motion state of the mesh to predict head motion. By employing ordinary differential equations (ODE) to model continuous dynamic sequences, it predicts head movements that closely resemble real head motion, making the animation more realistic and natural. Additionally, recognizing that the head motion generation given audio is not a one-to-one mapping problem, we introduce a noise module to help the head motion prediction module generate varied head motions given the same audio input. We also observed that previous facial animation methods primarily focus on generating vertices for the mouth region but use a single model to generate the entire face. This approach wastes some of the model’s fitting capacity on other regions. To solve this problem, we propose a cascaded mesh generation module that uses two modules to separately generate the vertex of mouth region and other facial regions. Extensive experiments and a perceptual user study show that our approach outperforms existing methods and produces relatively natural head motion.
Förderung
- National Key R&D Program of China
- National NSF of China
- Fundamental Research Funds for the Central Universities
- China Academy of Traditional Chinese Medicine Science and Technology Innovation Project
- Shandong Provincial Natural Science Foundation—Excellent Youth Program
Zitieren
APA
Sun, K., Zhang, J., Wang, W., Zhao, Y., & Sebe, N. (2026). Speech-Driven 3D Facial Animation with Natural Head Movements. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(7), 1-22. https://doi.org/10.1145/3811818
Vancouver
Sun K, Zhang J, Wang W, Zhao Y, Sebe N. Speech-Driven 3D Facial Animation with Natural Head Movements. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026;22(7):1-22. doi:10.1145/3811818
BibTeX
@article{Sun2026,
author = {Sun, Kuiyuan and Zhang, Jichao and Wang, Wei and Zhao, Yao and Sebe, Nicu},
title = {Speech-Driven 3D Facial Animation with Natural Head Movements},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2026},
volume = {22},
number = {7},
pages = {1--22},
doi = {10.1145/3811818}
}
EUVIMED