EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

TAMC: Textual Alignment and Masked Consistency for Open-Vocabulary 3D Scene Understanding

Juan Wang, Zhijie Wang iD, Tomo Miyazaki iD, Yaohou Fan iD, Shinichiro Omachi iD

Sensors · 24.09.2024 · 24 (19) · 6166

DOI: 10.3390/s24196166 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 24.09.2024. Verfasst von Juan Wang und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Communications Engineering, Graduate School of Engineering, Tohoku University, Sendai 9808579, Japan. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Japan Science and Technology Agency, JST SPRING.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Three-dimensional (3D) Scene Understanding achieves environmental perception by extracting and analyzing point cloud data with wide applications including virtual reality, robotics, etc. Previous methods align the 2D image feature from a pre-trained CLIP model and the 3D point cloud feature for the open vocabulary scene understanding ability. We believe that existing methods have the following two deficiencies: (1) the 3D feature extraction process ignores the challenges of real scenarios, i.e., point cloud data are very sparse and even incomplete; (2) the training stage lacks direct text supervision, leading to inconsistency with the inference stage. To address the first issue, we employ a Masked Consistency training policy. Specifically, during the alignment of 3D and 2D features, we mask some 3D features to force the model to understand the entire scene using only partial 3D features. For the second issue, we generate pseudo-text labels and align them with the 3D features during the training process. In particular, we first generate a description for each 2D image belonging to the same 3D scene and then use a summarization model to fuse these descriptions into a single description of the scene. Subsequently, we align 2D-3D features and 3D-text features simultaneously during training. Massive experiments demonstrate the effectiveness of our method, outperforming state-of-the-art approaches.

Schlagwörter

Stichwörter der Autor:innen

3D Scene UnderstandingMasked ConsistencyTextual Alignmentcontrastive learningmulti-modal learningopen vocabulary

Autor:innen und Institutionen

  • Department of Communications Engineering, Graduate School of Engineering, Tohoku University, Sendai 9808579, Japan.
  • RIKEN AIP, Tokyo 1030027, Japan.

ORCID-Profile: Zhijie Wang, Tomo Miyazaki, Yaohou Fan, Shinichiro Omachi

Förderung

  • Japan Science and Technology Agency
  • JST SPRING

Zitieren

APA

Wang, J., Wang, Z., Miyazaki, T., Fan, Y., & Omachi, S. (2024). TAMC: Textual Alignment and Masked Consistency for Open-Vocabulary 3D Scene Understanding. Sensors, 24(19), 6166. https://doi.org/10.3390/s24196166

Vancouver

Wang J, Wang Z, Miyazaki T, Fan Y, Omachi S. TAMC: Textual Alignment and Masked Consistency for Open-Vocabulary 3D Scene Understanding. Sensors. 2024;24(19):6166. doi:10.3390/s24196166

BibTeX
@article{Wang2024,
  author = {Wang, Juan and Wang, Zhijie and Miyazaki, Tomo and Fan, Yaohou and Omachi, Shinichiro},
  title = {TAMC: Textual Alignment and Masked Consistency for Open-Vocabulary 3D Scene Understanding},
  journal = {Sensors},
  year = {2024},
  volume = {24},
  number = {19},
  pages = {6166},
  doi = {10.3390/s24196166}
}
Kostenlos starten →