EUVIMEDEUVIMED entdecken
Crossref

Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style

Shuai Tan, Bin Ji, Ye Pan

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (5) · 5079-5087

DOI: 10.1609/aaai.v38i5.28313 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Shuai Tan und 2 weiteren Autor:innen. Die Arbeit wurde bisher 39-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Although automatically animating audio-driven talking heads has recently received growing interest, previous efforts have mainly concentrated on achieving lip synchronization with the audio, neglecting two crucial elements for generating expressive videos: emotion style and art style. In this paper, we present an innovative audio-driven talking face generation method called Style2Talker. It involves two stylized stages, namely Style-E and Style-A, which integrate text-controlled emotion style and picture-controlled art style into the final output. In order to prepare the scarce emotional text descriptions corresponding to the videos, we propose a labor-free paradigm that employs large-scale pretrained models to automatically annotate emotional text labels for existing audio-visual datasets. Incorporating the synthetic emotion texts, the Style-E stage utilizes a large-scale CLIP model to extract emotion representations, which are combined with the audio, serving as the condition for an efficient latent diffusion model designed to produce emotional motion coefficients of a 3DMM model. Moving on to the Style-A stage, we develop a coefficient-driven motion generator and an art-specific style path embedded in the well-known StyleGAN. This allows us to synthesize high-resolution artistically stylized talking head videos using the generated emotional motion coefficients and an art style source picture. Moreover, to better preserve image details and avoid artifacts, we provide StyleGAN with the multi-scale content features extracted from the identity image and refine its intermediate feature maps by the designed content encoder and refinement network, respectively. Extensive experimental results demonstrate our method outperforms existing state-of-the-art methods in terms of audio-lip synchronization and performance of both emotion style and art style.

Zitieren

APA

Tan, S., Ji, B., & Pan, Y. (2024). Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style. Proceedings of the AAAI Conference on Artificial Intelligence, 38(5), 5079-5087. https://doi.org/10.1609/aaai.v38i5.28313

Vancouver

Tan S, Ji B, Pan Y. Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(5):5079-5087. doi:10.1609/aaai.v38i5.28313

BibTeX
@article{Tan2024,
  author = {Tan, Shuai and Ji, Bin and Pan, Ye},
  title = {Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {5},
  pages = {5079--5087},
  doi = {10.1609/aaai.v38i5.28313}
}
Kostenlos starten →