StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024
DOI: 10.1109/tpami.2024.3357808 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. Verfasst von Suzhen Wang und 7 weiteren Autor:innen. Hauptthemen laut MEDLINE: Facial Expression, Speech, Video Recording, Head Movements. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 67 Literaturangaben. Gefördert von: 2022 Hangzhou Key Science and Technology Innovation, Key Research and Development Program of Zhejiang Province, ARC-Discovery.
Abstract (Originalsprache)
Individuals have unique facial expression and head pose styles that reflect their personalized speaking styles. Existing one-shot talking head methods cannot capture such personalized characteristics and therefore fail to produce diverse speaking styles in the final videos. To address this challenge, we propose a one-shot style-controllable talking face generation method that can obtain speaking styles from reference speaking videos and drive the one-shot portrait to speak with the reference speaking styles and another piece of audio. Our method aims to synthesize the style-controllable coefficients of a 3D Morphable Model (3DMM), including facial expressions and head movements, in a unified framework. Specifically, the proposed framework first leverages a style encoder to extract the desired speaking styles from the reference videos and transform them into style codes. Then, the framework uses a style-aware decoder to synthesize the coefficients of 3DMM from the audio input and style codes. During decoding, our framework adopts a two-branch architecture, which generates the stylized facial expression coefficients and stylized head movement coefficients, respectively. After obtaining the coefficients of 3DMM, an image renderer renders the expression coefficients into a specific person's talking-head video. Extensive experiments demonstrate that our method generates visually authentic talking head videos with diverse speaking styles from only one portrait image and an audio clip.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Förderung
- 2022 Hangzhou Key Science and Technology Innovation
- Key Research and Development Program of Zhejiang Province
- ARC-Discovery
- ARC-DECRA
- National Natural Science Foundation of China
- Institute Guo Qiang
- Tsinghua University
- SAIC Motor
Zitieren
APA
Wang, S., Ma, Y., Ding, Y., Hu, Z., Fan, C., Lv, T., Deng, Z., & Yu, X. (2024). StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2024.3357808
Vancouver
Wang S, Ma Y, Ding Y, Hu Z, Fan C, Lv T, et al. StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024. doi:10.1109/tpami.2024.3357808
BibTeX
@article{Wang2024,
author = {Wang, Suzhen and Ma, Yifeng and Ding, Yu and Hu, Zhipeng and Fan, Changjie and Lv, Tangjie and Deng, Zhidong and Yu, Xin},
title = {StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
doi = {10.1109/tpami.2024.3357808}
}
EUVIMED