Emotionally Controllable Audio-driven Talking Face Generation
ACM Transactions on Multimedia Computing, Communications, and Applications · 28.02.2026 · 22 (2) · 1-22
DOI: 10.1145/3779219 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 10.02.2026. Verfasst von Yifan Xu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 65 Literaturangaben. Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
Talking face generation is a technique that synthesizes realistic, speech-synchronized facial animations from static images driven by audio or textual input. In recent years, notable advancements have been achieved in the generation of realistic lip-synchronized talking face videos. Although these methods have demonstrated highly realistic results in lip-sync generation, they rarely focus on the expression of emotions, which is essential for achieving emotionally rich and lifelike talking face videos. In this article, we propose a novel emotionally controllable audio-driven talking face generation framework, termed ECATFG. Specifically, the proposed ECATFG mainly consists of three modules: template video generator, landmark generator, and rendering module. Firstly, we integrate a customized motion transfer module into StyleGAN to generate template videos that encapsulate the target emotions and head movements. Subsequently, a Mamba-based landmark generator is proposed to accurately model the correlation between input audio and facial landmarks. Afterwards, in the rendering module, we employ a framework that combines both AdaIN and SPADE layers to transform audio-predicted landmarks into highly realistic images, guided by reference images and facial contour landmarks. Finally, comprehensive experiments validate the effectiveness of ECATFG, demonstrating its capability to generate high-quality talking face videos that not only achieve exceptional lip-synchronization accuracy but also effectively convey a wide range of emotions.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Xu, Y., Zhao, S., Liu, S., Xu, T., & Chen, E. (2026). Emotionally Controllable Audio-driven Talking Face Generation. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(2), 1-22. https://doi.org/10.1145/3779219
Vancouver
Xu Y, Zhao S, Liu S, Xu T, Chen E. Emotionally Controllable Audio-driven Talking Face Generation. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026;22(2):1-22. doi:10.1145/3779219
BibTeX
@article{Xu2026,
author = {Xu, Yifan and Zhao, Sirui and Liu, Shifeng and Xu, Tong and Chen, Enhong},
title = {Emotionally Controllable Audio-driven Talking Face Generation},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2026},
volume = {22},
number = {2},
pages = {1--22},
doi = {10.1145/3779219}
}
EUVIMED