EUVIMEDEUVIMED entdecken
Crossref

StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads

Suzhen Wang, Yifeng Ma, Yu Ding, Zhipeng Hu, Changjie Fan, Tangjie Lv, Zhidong Deng, Xin Yu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024

DOI: 10.1109/tpami.2024.3357808 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. Verfasst von Suzhen Wang und 7 weiteren Autor:innen. Hauptthemen laut MEDLINE: Facial Expression, Speech, Video Recording, Head Movements. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 67 Literaturangaben. Gefördert von: 2022 Hangzhou Key Science and Technology Innovation, Key Research and Development Program of Zhejiang Province, ARC-Discovery.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Individuals have unique facial expression and head pose styles that reflect their personalized speaking styles. Existing one-shot talking head methods cannot capture such personalized characteristics and therefore fail to produce diverse speaking styles in the final videos. To address this challenge, we propose a one-shot style-controllable talking face generation method that can obtain speaking styles from reference speaking videos and drive the one-shot portrait to speak with the reference speaking styles and another piece of audio. Our method aims to synthesize the style-controllable coefficients of a 3D Morphable Model (3DMM), including facial expressions and head movements, in a unified framework. Specifically, the proposed framework first leverages a style encoder to extract the desired speaking styles from the reference videos and transform them into style codes. Then, the framework uses a style-aware decoder to synthesize the coefficients of 3DMM from the audio input and style codes. During decoding, our framework adopts a two-branch architecture, which generates the stylized facial expression coefficients and stylized head movement coefficients, respectively. After obtaining the coefficients of 3DMM, an image renderer renders the expression coefficients into a specific person's talking-head video. Extensive experiments demonstrate that our method generates visually authentic talking head videos with diverse speaking styles from only one portrait image and an audio clip.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Facial ExpressionSpeechVideo RecordingHead MovementsHumansAlgorithmsHeadImaging, Three-DimensionalFaceImage Processing, Computer-Assisted

Förderung

  • 2022 Hangzhou Key Science and Technology Innovation
  • Key Research and Development Program of Zhejiang Province
  • ARC-Discovery
  • ARC-DECRA
  • National Natural Science Foundation of China
  • Institute Guo Qiang
  • Tsinghua University
  • SAIC Motor

Zitieren

APA

Wang, S., Ma, Y., Ding, Y., Hu, Z., Fan, C., Lv, T., Deng, Z., & Yu, X. (2024). StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2024.3357808

Vancouver

Wang S, Ma Y, Ding Y, Hu Z, Fan C, Lv T, et al. StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024. doi:10.1109/tpami.2024.3357808

BibTeX
@article{Wang2024,
  author = {Wang, Suzhen and Ma, Yifeng and Ding, Yu and Hu, Zhipeng and Fan, Changjie and Lv, Tangjie and Deng, Zhidong and Yu, Xin},
  title = {StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  doi = {10.1109/tpami.2024.3357808}
}
Kostenlos starten →