NeuralRecon: Real-Time Coherent 3D Scene Reconstruction From Monocular Video
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-12 · 46 (12) · 7542-7555
DOI: 10.1109/tpami.2024.3393141 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-12. Verfasst von Xi Chen und 4 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 85 Literaturangaben. Gefördert von: National Key Research and Development Program of China, ZJU-SenseTime, CAD&CG.
Abstract (Originalsprache)
We present a novel framework named NeuralRecon for real-time 3D scene reconstruction from a monocular video. Unlike previous methods that estimate single-view depth maps separately on each key-frame and fuse them later, we propose to directly reconstruct local surfaces represented as sparse TSDF volumes for each video fragment sequentially by a neural network. A learning-based TSDF fusion module based on gated recurrent units is used to guide the network to fuse features from previous fragments. This design allows the network to capture local smoothness prior and global shape prior of 3D surfaces when sequentially reconstructing the surfaces, resulting in accurate, coherent, and real-time surface reconstruction. The fused features can also be used to predict semantic labels, allowing our method to reconstruct and segment the 3D scene simultaneously. Furthermore, we purpose an efficient self-supervised fine-tuning scheme that refines scene geometry based on input images through differentiable volume rendering. This fine-tuning scheme improves reconstruction quality on the fine-tuned scenes, as well as the generalization to similar test scenes. The experiments on ScanNet, 7-Scenes and Replica datasets show that our system outperforms state-of-the-art methods in terms of both accuracy and speed.
Förderung
- National Key Research and Development Program of China
- ZJU-SenseTime
- CAD&CG
- Zhejiang University
Zitieren
APA
Chen, X., Sun, J., Xie, Y., Bao, H., & Zhou, X. (2024). NeuralRecon: Real-Time Coherent 3D Scene Reconstruction From Monocular Video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 7542-7555. https://doi.org/10.1109/tpami.2024.3393141
Vancouver
Chen X, Sun J, Xie Y, Bao H, Zhou X. NeuralRecon: Real-Time Coherent 3D Scene Reconstruction From Monocular Video. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(12):7542-7555. doi:10.1109/tpami.2024.3393141
BibTeX
@article{Chen2024,
author = {Chen, Xi and Sun, Jiaming and Xie, Yiming and Bao, Hujun and Zhou, Xiaowei},
title = {NeuralRecon: Real-Time Coherent 3D Scene Reconstruction From Monocular Video},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
volume = {46},
number = {12},
pages = {7542--7555},
doi = {10.1109/tpami.2024.3393141}
}
EUVIMED