EUVIMEDEUVIMED entdecken
Crossref

Speed-Aware Audio-Driven Speech Animation using Adaptive Windows

Sunjin Jung, Yeongho Seol, Kwanggyoon Seo, Hyeonho Na, Seonghyeon Kim, Vanessa Tan, Junyong Noh

ACM Transactions on Graphics · 28.02.2025 · 44 (1) · 1-14

DOI: 10.1145/3691341 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Graphics, 2024-10. Verfasst von Sunjin Jung und 6 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 49 Literaturangaben. Gefördert von: Culture, Sports and Tourism R&D Program, Korea Creative Content Agency, Ministry of Culture, Sports and Tourism in 2023.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We present a novel method that can generate realistic speech animations of a 3D face from audio using multiple adaptive windows. In contrast to previous studies that use a fixed size audio window, our method accepts an adaptive audio window as input, reflecting the audio speaking rate to use consistent phonemic information. Our system consists of three parts. First, the speaking rate is estimated from the input audio using a neural network trained in a self-supervised manner. Second, the appropriate window size that encloses the audio features is predicted adaptively based on the estimated speaking rate. Another key element lies in the use of multiple audio windows of different sizes as input to the animation generator: a small window to concentrate on detailed information and a large window to consider broad phonemic information near the center frame. Finally, the speech animation is generated from the multiple adaptive audio windows. Our method can generate realistic speech animations from in-the-wild audios at any speaking rate, i.e., fast raps, slow songs, as well as normal speech. We demonstrate via extensive quantitative and qualitative evaluations including a user study that our method outperforms state-of-the-art approaches.

Förderung

  • Culture, Sports and Tourism R&D Program
  • Korea Creative Content Agency
  • Ministry of Culture, Sports and Tourism in 2023
  • Development of Universal Fashion Creation Platform Technology for Avatar Personality Expression

Zitieren

APA

Jung, S., Seol, Y., Seo, K., Na, H., Kim, S., Tan, V., & Noh, J. (2025). Speed-Aware Audio-Driven Speech Animation using Adaptive Windows. ACM Transactions on Graphics, 44(1), 1-14. https://doi.org/10.1145/3691341

Vancouver

Jung S, Seol Y, Seo K, Na H, Kim S, Tan V, et al. Speed-Aware Audio-Driven Speech Animation using Adaptive Windows. ACM Transactions on Graphics. 2025;44(1):1-14. doi:10.1145/3691341

BibTeX
@article{Jung2025,
  author = {Jung, Sunjin and Seol, Yeongho and Seo, Kwanggyoon and Na, Hyeonho and Kim, Seonghyeon and Tan, Vanessa and Noh, Junyong},
  title = {Speed-Aware Audio-Driven Speech Animation using Adaptive Windows},
  journal = {ACM Transactions on Graphics},
  year = {2025},
  volume = {44},
  number = {1},
  pages = {1--14},
  doi = {10.1145/3691341}
}
Kostenlos starten →