Speed-Aware Audio-Driven Speech Animation using Adaptive Windows
ACM Transactions on Graphics · 28.02.2025 · 44 (1) · 1-14
DOI: 10.1145/3691341 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Graphics, 2024-10. Verfasst von Sunjin Jung und 6 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 49 Literaturangaben. Gefördert von: Culture, Sports and Tourism R&D Program, Korea Creative Content Agency, Ministry of Culture, Sports and Tourism in 2023.
Abstract (Originalsprache)
We present a novel method that can generate realistic speech animations of a 3D face from audio using multiple adaptive windows. In contrast to previous studies that use a fixed size audio window, our method accepts an adaptive audio window as input, reflecting the audio speaking rate to use consistent phonemic information. Our system consists of three parts. First, the speaking rate is estimated from the input audio using a neural network trained in a self-supervised manner. Second, the appropriate window size that encloses the audio features is predicted adaptively based on the estimated speaking rate. Another key element lies in the use of multiple audio windows of different sizes as input to the animation generator: a small window to concentrate on detailed information and a large window to consider broad phonemic information near the center frame. Finally, the speech animation is generated from the multiple adaptive audio windows. Our method can generate realistic speech animations from in-the-wild audios at any speaking rate, i.e., fast raps, slow songs, as well as normal speech. We demonstrate via extensive quantitative and qualitative evaluations including a user study that our method outperforms state-of-the-art approaches.
Förderung
- Culture, Sports and Tourism R&D Program
- Korea Creative Content Agency
- Ministry of Culture, Sports and Tourism in 2023
- Development of Universal Fashion Creation Platform Technology for Avatar Personality Expression
Zitieren
APA
Jung, S., Seol, Y., Seo, K., Na, H., Kim, S., Tan, V., & Noh, J. (2025). Speed-Aware Audio-Driven Speech Animation using Adaptive Windows. ACM Transactions on Graphics, 44(1), 1-14. https://doi.org/10.1145/3691341
Vancouver
Jung S, Seol Y, Seo K, Na H, Kim S, Tan V, et al. Speed-Aware Audio-Driven Speech Animation using Adaptive Windows. ACM Transactions on Graphics. 2025;44(1):1-14. doi:10.1145/3691341
BibTeX
@article{Jung2025,
author = {Jung, Sunjin and Seol, Yeongho and Seo, Kwanggyoon and Na, Hyeonho and Kim, Seonghyeon and Tan, Vanessa and Noh, Junyong},
title = {Speed-Aware Audio-Driven Speech Animation using Adaptive Windows},
journal = {ACM Transactions on Graphics},
year = {2025},
volume = {44},
number = {1},
pages = {1--14},
doi = {10.1145/3691341}
}
EUVIMED