EUVIMEDEUVIMED entdecken
Crossref

NeuralRecon: Real-Time Coherent 3D Scene Reconstruction From Monocular Video

Xi Chen, Jiaming Sun, Yiming Xie, Hujun Bao, Xiaowei Zhou

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-12 · 46 (12) · 7542-7555

DOI: 10.1109/tpami.2024.3393141 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-12. Verfasst von Xi Chen und 4 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 85 Literaturangaben. Gefördert von: National Key Research and Development Program of China, ZJU-SenseTime, CAD&CG.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We present a novel framework named NeuralRecon for real-time 3D scene reconstruction from a monocular video. Unlike previous methods that estimate single-view depth maps separately on each key-frame and fuse them later, we propose to directly reconstruct local surfaces represented as sparse TSDF volumes for each video fragment sequentially by a neural network. A learning-based TSDF fusion module based on gated recurrent units is used to guide the network to fuse features from previous fragments. This design allows the network to capture local smoothness prior and global shape prior of 3D surfaces when sequentially reconstructing the surfaces, resulting in accurate, coherent, and real-time surface reconstruction. The fused features can also be used to predict semantic labels, allowing our method to reconstruct and segment the 3D scene simultaneously. Furthermore, we purpose an efficient self-supervised fine-tuning scheme that refines scene geometry based on input images through differentiable volume rendering. This fine-tuning scheme improves reconstruction quality on the fine-tuned scenes, as well as the generalization to similar test scenes. The experiments on ScanNet, 7-Scenes and Replica datasets show that our system outperforms state-of-the-art methods in terms of both accuracy and speed.

Förderung

  • National Key Research and Development Program of China
  • ZJU-SenseTime
  • CAD&CG
  • Zhejiang University

Zitieren

APA

Chen, X., Sun, J., Xie, Y., Bao, H., & Zhou, X. (2024). NeuralRecon: Real-Time Coherent 3D Scene Reconstruction From Monocular Video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 7542-7555. https://doi.org/10.1109/tpami.2024.3393141

Vancouver

Chen X, Sun J, Xie Y, Bao H, Zhou X. NeuralRecon: Real-Time Coherent 3D Scene Reconstruction From Monocular Video. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(12):7542-7555. doi:10.1109/tpami.2024.3393141

BibTeX
@article{Chen2024,
  author = {Chen, Xi and Sun, Jiaming and Xie, Yiming and Bao, Hujun and Zhou, Xiaowei},
  title = {NeuralRecon: Real-Time Coherent 3D Scene Reconstruction From Monocular Video},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {12},
  pages = {7542--7555},
  doi = {10.1109/tpami.2024.3393141}
}
Kostenlos starten →