Semantic Gaussians: Open-Vocabulary Scene Understanding With 3D Gaussian Splatting
IEEE Transactions on Circuits and Systems for Video Technology · 2026-07 · 36 (7) · 9265-9277
DOI: 10.1109/tcsvt.2026.3675320 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2026-07. Verfasst von Jun Guo und 4 weiteren Autor:innen. Die Arbeit wurde bisher 27-mal zitiert (Crossref) und nennt 80 Literaturangaben. Gefördert von: National Natural Science Fund, Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China, State Key Laboratory of General Artificial Intelligence (BIGAI) and Tsinghua University, Beijing, China.
Abstract (Originalsprache)
Open-vocabulary 3D scene understanding presents a significant challenge in computer vision, with wide-ranging applications in embodied agents and augmented reality systems. Existing methods adopt neural rendering methods as 3D representations and jointly optimize color and semantic features to achieve rendering and scene understanding simultaneously. In this paper, we introduce Semantic Gaussians, a novel open-vocabulary scene understanding approach based on 3D Gaussian Splatting. Our key idea is to distill knowledge from 2D pre-trained models to 3D Gaussians. Unlike existing methods, we design a versatile projection approach that maps various 2D semantic features from pre-trained image encoders into a novel semantic component of 3D Gaussians, which is based on spatial relationships and needs no additional training. We further build a 3D semantic network that directly predicts the semantic component from raw 3D Gaussians for fast inference. The quantitative results on ScanNet segmentation and LERF object localization demonstrate the superior performance of our method. Additionally, we explore several applications of Semantic Gaussians, including object part segmentation, instance segmentation, scene editing, and spatiotemporal segmentation, with better qualitative results over 2D and 3D baselines, highlighting its versatility and effectiveness in supporting diverse downstream tasks.
Förderung
- National Natural Science Fund
- Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China
- State Key Laboratory of General Artificial Intelligence (BIGAI) and Tsinghua University, Beijing, China
Zitieren
APA
Guo, J., Ma, X., Fan, Y., Liu, H., & Li, Q. (2026). Semantic Gaussians: Open-Vocabulary Scene Understanding With 3D Gaussian Splatting. IEEE Transactions on Circuits and Systems for Video Technology, 36(7), 9265-9277. https://doi.org/10.1109/tcsvt.2026.3675320
Vancouver
Guo J, Ma X, Fan Y, Liu H, Li Q. Semantic Gaussians: Open-Vocabulary Scene Understanding With 3D Gaussian Splatting. IEEE Transactions on Circuits and Systems for Video Technology. 2026;36(7):9265-9277. doi:10.1109/tcsvt.2026.3675320
BibTeX
@article{Guo2026,
author = {Guo, Jun and Ma, Xiaojian and Fan, Yue and Liu, Huaping and Li, Qing},
title = {Semantic Gaussians: Open-Vocabulary Scene Understanding With 3D Gaussian Splatting},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2026},
volume = {36},
number = {7},
pages = {9265--9277},
doi = {10.1109/tcsvt.2026.3675320}
}
EUVIMED