EUVIMEDEUVIMED entdecken
Crossref

FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding

Xingxing Zuo, Pouya Samangouei, Yunwen Zhou, Yan Di, Mingyang Li

International Journal of Computer Vision · 2025-02 · 133 (2) · 611-627

DOI: 10.1007/s11263-024-02183-8 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 12.08.2024. Verfasst von Xingxing Zuo und 4 weiteren Autor:innen. Die Arbeit wurde bisher 83-mal zitiert (Crossref) und nennt 66 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Precisely perceiving the geometric and semantic properties of real-world 3D objects is crucial for the continued evolution of augmented reality and robotic applications. To this end, we present Foundation Model Embedded Gaussian Splatting (FMGS), which incorporates vision-language embeddings of foundation models into 3D Gaussian Splatting (GS). The key contribution of this work is an efficient method to reconstruct and represent 3D vision-language models. This is achieved by distilling feature maps generated from image-based foundation models into those rendered from our 3D model. To ensure high-quality rendering and fast training, we introduce a novel scene representation by integrating strengths from both GS and multi-resolution hash encodings (MHE). Our effective training procedure also introduces a pixel alignment loss that makes the rendered feature distance of same semantic entities close, following the pixel-level semantic boundaries. Our results demonstrate remarkable multi-view semantic consistency, facilitating diverse downstream tasks, beating state-of-the-art methods by 10.2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${10.2}$$\end{document}object detection, despite that we are 851×\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${851\times }$$\end{document} faster for inference. This research explores the intersection of vision, language, and 3D scene representation, paving the way for enhanced scene understanding in uncontrolled real-world environments. We plan to release the code on the [project page].

Zitieren

APA

Zuo, X., Samangouei, P., Zhou, Y., Di, Y., & Li, M. (2025). FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding. International Journal of Computer Vision, 133(2), 611-627. https://doi.org/10.1007/s11263-024-02183-8

Vancouver

Zuo X, Samangouei P, Zhou Y, Di Y, Li M. FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding. International Journal of Computer Vision. 2025;133(2):611-627. doi:10.1007/s11263-024-02183-8

BibTeX
@article{Zuo2025,
  author = {Zuo, Xingxing and Samangouei, Pouya and Zhou, Yunwen and Di, Yan and Li, Mingyang},
  title = {FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding},
  journal = {International Journal of Computer Vision},
  year = {2025},
  volume = {133},
  number = {2},
  pages = {611--627},
  doi = {10.1007/s11263-024-02183-8}
}
Kostenlos starten →