EUVIMEDEUVIMED entdecken
Crossref

Emotionally Controllable Audio-driven Talking Face Generation

Yifan Xu, Sirui Zhao, Shifeng Liu, Tong Xu, Enhong Chen

ACM Transactions on Multimedia Computing, Communications, and Applications · 28.02.2026 · 22 (2) · 1-22

DOI: 10.1145/3779219 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 10.02.2026. Verfasst von Yifan Xu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 65 Literaturangaben. Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Talking face generation is a technique that synthesizes realistic, speech-synchronized facial animations from static images driven by audio or textual input. In recent years, notable advancements have been achieved in the generation of realistic lip-synchronized talking face videos. Although these methods have demonstrated highly realistic results in lip-sync generation, they rarely focus on the expression of emotions, which is essential for achieving emotionally rich and lifelike talking face videos. In this article, we propose a novel emotionally controllable audio-driven talking face generation framework, termed ECATFG. Specifically, the proposed ECATFG mainly consists of three modules: template video generator, landmark generator, and rendering module. Firstly, we integrate a customized motion transfer module into StyleGAN to generate template videos that encapsulate the target emotions and head movements. Subsequently, a Mamba-based landmark generator is proposed to accurately model the correlation between input audio and facial landmarks. Afterwards, in the rendering module, we employ a framework that combines both AdaIN and SPADE layers to transform audio-predicted landmarks into highly realistic images, guided by reference images and facial contour landmarks. Finally, comprehensive experiments validate the effectiveness of ECATFG, demonstrating its capability to generate high-quality talking face videos that not only achieve exceptional lip-synchronization accuracy but also effectively convey a wide range of emotions.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Xu, Y., Zhao, S., Liu, S., Xu, T., & Chen, E. (2026). Emotionally Controllable Audio-driven Talking Face Generation. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(2), 1-22. https://doi.org/10.1145/3779219

Vancouver

Xu Y, Zhao S, Liu S, Xu T, Chen E. Emotionally Controllable Audio-driven Talking Face Generation. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026;22(2):1-22. doi:10.1145/3779219

BibTeX
@article{Xu2026,
  author = {Xu, Yifan and Zhao, Sirui and Liu, Shifeng and Xu, Tong and Chen, Enhong},
  title = {Emotionally Controllable Audio-driven Talking Face Generation},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2026},
  volume = {22},
  number = {2},
  pages = {1--22},
  doi = {10.1145/3779219}
}
Kostenlos starten →