EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Audio-driven 3D Facial Animation with Multimodal Context Enhancement

Yanlei Lei, Jing Chi, Chunlin Zhao, Minfeng Xu, Caiming Zhang

ACM Transactions on Multimedia Computing, Communications, and Applications · 30.11.2026 · 22 (11) · 1-23

DOI: 10.1145/3856308 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 11.11.2026. Verfasst von Yanlei Lei und 4 weiteren Autor:innen. Die Arbeit nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Key Research and Development project of Shandong Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Audio-driven 3D facial animation has emerged as a prominent research topic in computer graphics. Existing autoregressive methods are often slow and suffer from error accumulation, whereas non-autoregressive methods often struggle to establish global contextual dependencies. Moreover, current approaches remain insufficient in the cross-modal semantic alignment between audio and facial animation. To address these challenges, we propose a non-autoregressive audio-driven 3D facial animation method with multimodal context enhancement. Our method fully exploits multi-scale facial representations as well as the global contextual information within facial and audio-feature sequences, enabling deep cross-modal fusion and precise semantic alignment. Specifically, we develop a global–local encoder built with spiral convolution, which facilitates complementary learning of global basic features and local fine-grained details, thereby improving the quality of facial feature extraction. We further design a multimodal context-enhanced cross-modal modulation module. This module adopts an Adaptive Instance Normalization (AdaIN)-based cross-modal fusion strategy to modulate facial features with audio sequence context in a single step, achieving fast generation without error accumulation. In addition, the module constructs a feature region cross-attention mechanism to capture contextual dependencies within facial features, and leverages globally-aware adaptive channel weighting to model temporal contextual relations of multimodal fused features. These designs enable effective exploitation of global context and substantially improve cross-modal semantic alignment accuracy. Extensive experiments demonstrate that our method outperforms mainstream baselines in both animation quality and generation efficiency.

Förderung

  • National Natural Science Foundation of China
  • Key Research and Development project of Shandong Province

Zitieren

APA

Lei, Y., Chi, J., Zhao, C., Xu, M., & Zhang, C. (2026). Audio-driven 3D Facial Animation with Multimodal Context Enhancement. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(11), 1-23. https://doi.org/10.1145/3856308

Vancouver

Lei Y, Chi J, Zhao C, Xu M, Zhang C. Audio-driven 3D Facial Animation with Multimodal Context Enhancement. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026;22(11):1-23. doi:10.1145/3856308

BibTeX
@article{Lei2026,
  author = {Lei, Yanlei and Chi, Jing and Zhao, Chunlin and Xu, Minfeng and Zhang, Caiming},
  title = {Audio-driven 3D Facial Animation with Multimodal Context Enhancement},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2026},
  volume = {22},
  number = {11},
  pages = {1--23},
  doi = {10.1145/3856308}
}
Kostenlos starten →