EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

360-degree Human Video Generation with 4D Diffusion Transformer

Ruizhi Shao, Youxin Pang, Zerong Zheng, Jingxiang Sun, Yebin Liu

ACM Transactions on Graphics · 19.12.2024 · 43 (6) · 1-13

DOI: 10.1145/3687980 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 19.11.2024. Verfasst von Ruizhi Shao und 4 weiteren Autor:innen. Die Arbeit wurde bisher 37-mal zitiert (Crossref) und nennt 54 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We present a novel approach for generating 360-degree high-quality, spatiotemporally coherent human videos from a single image. Our framework combines the strengths of diffusion transformers for capturing global correlations across viewpoints and time, and CNNs for accurate condition injection. The core is a hierarchical 4D transformer architecture that factorizes self-attention across views, time steps, and spatial dimensions, enabling efficient modeling of the 4D space. Precise conditioning is achieved by injecting human identity, camera parameters, and temporal signals into the respective transformers. To train this model, we collect a multi-dimensional dataset spanning images, videos, multi-view data, and limited 4D footage, along with a tailored multi-dimensional training strategy. Our approach overcomes the limitations of previous methods based on generative adversarial networks or vanilla diffusion models, which struggle with complex motions, viewpoint changes, and generalization. Through extensive experiments, we demonstrate our method's ability to synthesize 360-degree realistic, coherent human motion videos, paving the way for advanced multimedia applications in areas such as virtual reality and animation.

Zitieren

APA

Shao, R., Pang, Y., Zheng, Z., Sun, J., & Liu, Y. (2024). 360-degree Human Video Generation with 4D Diffusion Transformer. ACM Transactions on Graphics, 43(6), 1-13. https://doi.org/10.1145/3687980

Vancouver

Shao R, Pang Y, Zheng Z, Sun J, Liu Y. 360-degree Human Video Generation with 4D Diffusion Transformer. ACM Transactions on Graphics. 2024;43(6):1-13. doi:10.1145/3687980

BibTeX
@article{Shao2024,
  author = {Shao, Ruizhi and Pang, Youxin and Zheng, Zerong and Sun, Jingxiang and Liu, Yebin},
  title = {360-degree Human Video Generation with 4D Diffusion Transformer},
  journal = {ACM Transactions on Graphics},
  year = {2024},
  volume = {43},
  number = {6},
  pages = {1--13},
  doi = {10.1145/3687980}
}
Kostenlos starten →