EUVIMEDEUVIMED entdecken
Crossref

FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for Audio-Driven Portrait Animation

Mengchao Wang, Wang Qiang, Fan Jiang, Mu Xu

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (12) · 9966-9974

DOI: 10.1609/aaai.v40i12.37962 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Mengchao Wang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent advances in audio-driven portrait animation have demonstrated impressive capabilities. However, existing methods struggle to align with fine-grained human preferences across multiple dimensions, such as motion naturalness, lip-sync accuracy, and visual quality. This is due to the difficulty of optimizing among competing preference objectives, which often conflict with one another, and the scarcity of large-scale, high-quality datasets with multidimensional preference annotations. To address these, we first introduce Talking-Critic, a multimodal reward model that learns human-aligned reward functions to quantify how well generated videos satisfy multidimensional expectations. Leveraging this model, we curate Talking-NSQ, a large-scale multidimensional human preference dataset containing 410K preference pairs. Finally, we propose Timestep-Layer adaptive multi-expert Preference Optimization (TLPO), a novel framework for aligning diffusion-based portrait animation models with fine-grained, multidimensional preferences. TLPO decouples preferences into specialized expert modules, which are then fused across timesteps and network layers, enabling comprehensive, fine-grained enhancement across all dimensions without mutual interference. Experiments demonstrate that Talking-Critic significantly outperforms existing methods in aligning with human preference ratings. Meanwhile, TLPO achieves substantial improvements over baseline models in lip-sync accuracy, motion naturalness, and visual quality, exhibiting superior performance in both qualitative and quantitative evaluations.

Zitieren

APA

Wang, M., Qiang, W., Jiang, F., & Xu, M. (2026). FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for Audio-Driven Portrait Animation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(12), 9966-9974. https://doi.org/10.1609/aaai.v40i12.37962

Vancouver

Wang M, Qiang W, Jiang F, Xu M. FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for Audio-Driven Portrait Animation. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(12):9966-9974. doi:10.1609/aaai.v40i12.37962

BibTeX
@article{Wang2026,
  author = {Wang, Mengchao and Qiang, Wang and Jiang, Fan and Xu, Mu},
  title = {FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for Audio-Driven Portrait Animation},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {12},
  pages = {9966--9974},
  doi = {10.1609/aaai.v40i12.37962}
}
Kostenlos starten →