360-degree Human Video Generation with 4D Diffusion Transformer
ACM Transactions on Graphics · 19.12.2024 · 43 (6) · 1-13
DOI: 10.1145/3687980 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 19.11.2024. Verfasst von Ruizhi Shao und 4 weiteren Autor:innen. Die Arbeit wurde bisher 37-mal zitiert (Crossref) und nennt 54 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
We present a novel approach for generating 360-degree high-quality, spatiotemporally coherent human videos from a single image. Our framework combines the strengths of diffusion transformers for capturing global correlations across viewpoints and time, and CNNs for accurate condition injection. The core is a hierarchical 4D transformer architecture that factorizes self-attention across views, time steps, and spatial dimensions, enabling efficient modeling of the 4D space. Precise conditioning is achieved by injecting human identity, camera parameters, and temporal signals into the respective transformers. To train this model, we collect a multi-dimensional dataset spanning images, videos, multi-view data, and limited 4D footage, along with a tailored multi-dimensional training strategy. Our approach overcomes the limitations of previous methods based on generative adversarial networks or vanilla diffusion models, which struggle with complex motions, viewpoint changes, and generalization. Through extensive experiments, we demonstrate our method's ability to synthesize 360-degree realistic, coherent human motion videos, paving the way for advanced multimedia applications in areas such as virtual reality and animation.
Zitieren
APA
Shao, R., Pang, Y., Zheng, Z., Sun, J., & Liu, Y. (2024). 360-degree Human Video Generation with 4D Diffusion Transformer. ACM Transactions on Graphics, 43(6), 1-13. https://doi.org/10.1145/3687980
Vancouver
Shao R, Pang Y, Zheng Z, Sun J, Liu Y. 360-degree Human Video Generation with 4D Diffusion Transformer. ACM Transactions on Graphics. 2024;43(6):1-13. doi:10.1145/3687980
BibTeX
@article{Shao2024,
author = {Shao, Ruizhi and Pang, Youxin and Zheng, Zerong and Sun, Jingxiang and Liu, Yebin},
title = {360-degree Human Video Generation with 4D Diffusion Transformer},
journal = {ACM Transactions on Graphics},
year = {2024},
volume = {43},
number = {6},
pages = {1--13},
doi = {10.1145/3687980}
}
EUVIMED