EUVIMEDEUVIMED entdecken
Crossref

SceneDreamer: Unbounded 3D Scene Generation From 2D Image Collections

Zhaoxi Chen, Guangcong Wang, Ziwei Liu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2023-12 · 45 (12) · 15562-15576

DOI: 10.1109/tpami.2023.3321857 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023-12. Verfasst von Zhaoxi Chen und 2 weiteren Autor:innen. Die Arbeit wurde bisher 52-mal zitiert (Crossref) und nennt 76 Literaturangaben. Gefördert von: National Research Foundation Singapore, AI Singapore Programme, NTU NAP.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this work, we present SceneDreamer, an unconditional generative model for unbounded 3D scenes, which synthesizes large-scale 3D landscapes from random noise. Our framework is learned from in-the-wild 2D image collections only, without any 3D annotations. At the core of SceneDreamer is a principled learning paradigm comprising: 1) an efficient yet expressive 3D scene representation, 2) a generative scene parameterization, and 3) an effective renderer that can leverage the knowledge from 2D images. Our approach begins with an efficient bird's-eye-view (BEV) representation generated from simplex noise, which includes a height field for surface elevation and a semantic field for detailed scene semantics. This BEV scene representation enables: 1) representing a 3D scene with quadratic complexity, 2) disentangled geometry and semantics, and 3) efficient training. Moreover, we propose a novel generative neural hash grid to parameterize the latent space based on 3D positions and scene semantics, aiming to encode generalizable features across various scenes. Lastly, a neural volumetric renderer, learned from 2D image collections through adversarial training, is employed to produce photorealistic images. Extensive experiments demonstrate the effectiveness of SceneDreamer and superiority over state-of-the-art methods in generating vivid yet diverse unbounded 3D worlds.

Förderung

  • National Research Foundation Singapore
  • AI Singapore Programme
  • NTU NAP
  • MOE AcRF Tier 2
  • RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative

Zitieren

APA

Chen, Z., Wang, G., & Liu, Z. (2023). SceneDreamer: Unbounded 3D Scene Generation From 2D Image Collections. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(12), 15562-15576. https://doi.org/10.1109/tpami.2023.3321857

Vancouver

Chen Z, Wang G, Liu Z. SceneDreamer: Unbounded 3D Scene Generation From 2D Image Collections. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(12):15562-15576. doi:10.1109/tpami.2023.3321857

BibTeX
@article{Chen2023,
  author = {Chen, Zhaoxi and Wang, Guangcong and Liu, Ziwei},
  title = {SceneDreamer: Unbounded 3D Scene Generation From 2D Image Collections},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2023},
  volume = {45},
  number = {12},
  pages = {15562--15576},
  doi = {10.1109/tpami.2023.3321857}
}
Kostenlos starten →