EUVIMEDEUVIMED entdecken
Crossref

StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles

Yifeng Ma, Suzhen Wang, Zhipeng Hu, Changjie Fan, Tangjie Lv, Yu Ding, Zhidong Deng, Xin Yu

Proceedings of the AAAI Conference on Artificial Intelligence · 26.06.2023 · 37 (2) · 1896-1904

DOI: 10.1609/aaai.v37i2.25280 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 26.06.2023. Verfasst von Yifeng Ma und 7 weiteren Autor:innen. Die Arbeit wurde bisher 87-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Different people speak with diverse personalized speaking styles. Although existing one-shot talking head methods have made significant progress in lip sync, natural facial expressions, and stable head motions, they still cannot generate diverse speaking styles in the final talking head videos. To tackle this problem, we propose a one-shot style-controllable talking face generation framework. In a nutshell, we aim to attain a speaking style from an arbitrary reference speaking video and then drive the one-shot portrait to speak with the reference speaking style and another piece of audio. Specifically, we first develop a style encoder to extract dynamic facial motion patterns of a style reference video and then encode them into a style code. Afterward, we introduce a style-controllable decoder to synthesize stylized facial animations from the speech content and style code. In order to integrate the reference speaking style into generated videos, we design a style-aware adaptive transformer, which enables the encoded style code to adjust the weights of the feed-forward layers accordingly. Thanks to the style-aware adaptation mechanism, the reference speaking style can be better embedded into synthesized videos during decoding. Extensive experiments demonstrate that our method is capable of generating talking head videos with diverse speaking styles from only one portrait image and an audio clip while achieving authentic visual effects. Project Page: https://github.com/FuxiVirtualHuman/styletalk.

Zitieren

APA

Ma, Y., Wang, S., Hu, Z., Fan, C., Lv, T., Ding, Y., Deng, Z., & Yu, X. (2023). StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles. Proceedings of the AAAI Conference on Artificial Intelligence, 37(2), 1896-1904. https://doi.org/10.1609/aaai.v37i2.25280

Vancouver

Ma Y, Wang S, Hu Z, Fan C, Lv T, Ding Y, et al. StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles. Proceedings of the AAAI Conference on Artificial Intelligence. 2023;37(2):1896-1904. doi:10.1609/aaai.v37i2.25280

BibTeX
@article{Ma2023,
  author = {Ma, Yifeng and Wang, Suzhen and Hu, Zhipeng and Fan, Changjie and Lv, Tangjie and Ding, Yu and Deng, Zhidong and Yu, Xin},
  title = {StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2023},
  volume = {37},
  number = {2},
  pages = {1896--1904},
  doi = {10.1609/aaai.v37i2.25280}
}
Kostenlos starten →