SceneDreamer: Unbounded 3D Scene Generation From 2D Image Collections
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2023-12 · 45 (12) · 15562-15576
DOI: 10.1109/tpami.2023.3321857 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023-12. Verfasst von Zhaoxi Chen und 2 weiteren Autor:innen. Die Arbeit wurde bisher 52-mal zitiert (Crossref) und nennt 76 Literaturangaben. Gefördert von: National Research Foundation Singapore, AI Singapore Programme, NTU NAP.
Abstract (Originalsprache)
In this work, we present SceneDreamer, an unconditional generative model for unbounded 3D scenes, which synthesizes large-scale 3D landscapes from random noise. Our framework is learned from in-the-wild 2D image collections only, without any 3D annotations. At the core of SceneDreamer is a principled learning paradigm comprising: 1) an efficient yet expressive 3D scene representation, 2) a generative scene parameterization, and 3) an effective renderer that can leverage the knowledge from 2D images. Our approach begins with an efficient bird's-eye-view (BEV) representation generated from simplex noise, which includes a height field for surface elevation and a semantic field for detailed scene semantics. This BEV scene representation enables: 1) representing a 3D scene with quadratic complexity, 2) disentangled geometry and semantics, and 3) efficient training. Moreover, we propose a novel generative neural hash grid to parameterize the latent space based on 3D positions and scene semantics, aiming to encode generalizable features across various scenes. Lastly, a neural volumetric renderer, learned from 2D image collections through adversarial training, is employed to produce photorealistic images. Extensive experiments demonstrate the effectiveness of SceneDreamer and superiority over state-of-the-art methods in generating vivid yet diverse unbounded 3D worlds.
Förderung
- National Research Foundation Singapore
- AI Singapore Programme
- NTU NAP
- MOE AcRF Tier 2
- RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative
Zitieren
APA
Chen, Z., Wang, G., & Liu, Z. (2023). SceneDreamer: Unbounded 3D Scene Generation From 2D Image Collections. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(12), 15562-15576. https://doi.org/10.1109/tpami.2023.3321857
Vancouver
Chen Z, Wang G, Liu Z. SceneDreamer: Unbounded 3D Scene Generation From 2D Image Collections. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(12):15562-15576. doi:10.1109/tpami.2023.3321857
BibTeX
@article{Chen2023,
author = {Chen, Zhaoxi and Wang, Guangcong and Liu, Ziwei},
title = {SceneDreamer: Unbounded 3D Scene Generation From 2D Image Collections},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2023},
volume = {45},
number = {12},
pages = {15562--15576},
doi = {10.1109/tpami.2023.3321857}
}
EUVIMED