Scale-aware monocular reconstruction via robot kinematics and visual data in neural radiance fields
Artificial Intelligence Surgery · 15.08.2024 · 4 (3) · 187-98
DOI: 10.20517/ais.2024.12 ↗Kurz gefasst
Fachartikel in der Zeitschrift Artificial Intelligence Surgery, online erschienen am 15.08.2024. Verfasst von Ruofeng Wei und 6 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 38 Literaturangaben.
Abstract (Originalsprache)
Aim
Scale-aware 3D reconstruction of the surgical scene from a monocular endoscope is important for automatic navigation systems in robot-assisted surgery. However, traditional multi-view stereo methods purely utilize monocular images, which can recover 3D structures arbitrarily scaled with the real world. Current deep learning-based approaches rely on large training data for relative depth estimation and further 3D reconstruction with no scale. Inspired by recently proposed neural radiance fields (NeRF), we present a novel pipeline, KV-EndoNeRF, which explores limited multi-modal data (i.e., robot kinematics, and monocular endoscope) for surgical scene reconstruction with absolute scale.
Methods
We first extract scale information from robot kinematics data and then integrate it into sparse depth recovered from structure from motion (SfM). Based on the sparse depth supervision, we adapt a monocular depth estimation network to the current surgical scene to obtain scene-specific coarse depth. After adjusting the scale of coarse depth, we use it to guide the optimization of NeRF, resulting in absolute depth estimation. The 3D models of the tissue surface with real scale are recovered by fusing fine depth maps.
Results
Experimental results on the Stereo Correspondence And Reconstruction of Endoscopic Data (SCARED) demonstrate that KV-EndoNeRF excels in learning an absolute scale from robot kinematics and achieves 3D reconstruction with rich details of surface texture and high accuracy, outperforming other existing reconstruction methods.
Conclusion
Combining multi-modal image data with NeRF-based optimization represents a potential approach to achieve scale-aware 3D reconstruction of monocular endoscopic scenes.
Zitieren
APA
Wei, R., Guo, J., Lu, Y., Zhong, F., Liu, Y., Sun, D., & Dou, Q. (2024). Scale-aware monocular reconstruction via robot kinematics and visual data in neural radiance fields. Artificial Intelligence Surgery, 4(3), 187-98. https://doi.org/10.20517/ais.2024.12
Vancouver
Wei R, Guo J, Lu Y, Zhong F, Liu Y, Sun D, et al. Scale-aware monocular reconstruction via robot kinematics and visual data in neural radiance fields. Artificial Intelligence Surgery. 2024;4(3):187-98. doi:10.20517/ais.2024.12
BibTeX
@article{Wei2024,
author = {Wei, Ruofeng and Guo, Jiaxin and Lu, Yiang and Zhong, Fangxun and Liu, Yunhui and Sun, Dong and Dou, Qi},
title = {Scale-aware monocular reconstruction via robot kinematics and visual data in neural radiance fields},
journal = {Artificial Intelligence Surgery},
year = {2024},
volume = {4},
number = {3},
pages = {187--98},
doi = {10.20517/ais.2024.12}
}
EUVIMED