PERF: Panoramic Neural Radiance Field From a Single Panorama
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-10 · 46 (10) · 6905-6918
DOI: 10.1109/tpami.2024.3387307 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-10. Verfasst von Guangcong Wang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 42-mal zitiert (Crossref) und nennt 72 Literaturangaben. Gefördert von: National Research Foundation Singapore, NTU NAP, MOE AcRF Tier 2.
Abstract (Originalsprache)
Neural Radiance Field (NeRF) has achieved substantial progress in novel view synthesis given multi-view images. Recently, some works have attempted to train a NeRF from a single image with 3D priors. They mainly focus on a limited field of view with a few occlusions, which greatly limits their scalability to real-world 360-degree panoramic scenarios with large-size occlusions. In this paper, we present PERF, a 360-degree novel view synthesis framework that trains a panoramic neural radiance field from a single panorama. Notably, PERF allows 3D roaming in a complex scene without expensive and tedious image collection. To achieve this goal, we propose a novel collaborative RGBD inpainting method and a progressive inpainting-and-erasing method to lift up a 360-degree 2D scene to a 3D scene. Specifically, we first predict a panoramic depth map as initialization given a single panorama and reconstruct visible 3D regions with volume rendering. Then we introduce a collaborative RGBD inpainting approach into a NeRF for completing RGB images and depth maps from random views, which is derived from an RGB Stable Diffusion model and a monocular depth estimator. Finally, we introduce an inpainting-and-erasing strategy to avoid inconsistent geometry between a newly-sampled view and reference views. The two components are integrated into the learning of NeRFs in a unified optimization framework and achieve promising results. Extensive experiments on Replica and a new dataset PERF-in-the-wild demonstrate the superiority of our PERF over state-of-the-art methods. Our PERF can be widely used for real-world applications, such as panorama-to-3D, text-to-3D, and 3D scene stylization applications.
Förderung
- National Research Foundation Singapore
- NTU NAP
- MOE AcRF Tier 2
- Industry Alignment Fund-Industry Collaboration
Zitieren
APA
Wang, G., Wang, P., Chen, Z., Wang, W., Loy, C. C., & Liu, Z. (2024). PERF: Panoramic Neural Radiance Field From a Single Panorama. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10), 6905-6918. https://doi.org/10.1109/tpami.2024.3387307
Vancouver
Wang G, Wang P, Chen Z, Wang W, Loy CC, Liu Z. PERF: Panoramic Neural Radiance Field From a Single Panorama. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(10):6905-6918. doi:10.1109/tpami.2024.3387307
BibTeX
@article{Wang2024,
author = {Wang, Guangcong and Wang, Peng and Chen, Zhaoxi and Wang, Wenping and Loy, Chen Change and Liu, Ziwei},
title = {PERF: Panoramic Neural Radiance Field From a Single Panorama},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
volume = {46},
number = {10},
pages = {6905--6918},
doi = {10.1109/tpami.2024.3387307}
}
EUVIMED