EUVIMEDEUVIMED entdecken
Crossref

Grounding Vision and Language to 3D Masks for Long-Horizon Box Rearrangement

Ashish Malik, Caleb Lowe, Aayam Shrestha, Stefan Lee, Fuxin Li, Alan Fern

Proceedings of the International Conference on Automated Planning and Scheduling · 08.06.2026 · 36 (1) · 729-737

DOI: 10.1609/icaps.v36i1.42893 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the International Conference on Automated Planning and Scheduling, online erschienen am 08.06.2026. Verfasst von Ashish Malik und 5 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We study long-horizon planning in 3D environments from under-specified natural-language goals using only visual observations, focusing on multi-step 3D box rearrangement tasks. Existing approaches typically rely on symbolic planners with brittle relational grounding of states and goals, or on direct action-sequence generation from 2D vision-language models (VLMs). Both approaches struggle with reasoning over many objects, rich 3D geometry, and implicit semantic constraints. Recent advances in 3D VLMs demonstrate strong grounding of natural-language referents to 3D segmentation masks, suggesting the potential for more general planning capabilities. We extend existing 3D grounding models and propose Reactive Action Mask Planner (RAMP-3D), which formulates long-horizon planning as sequential reactive prediction of paired 3D masks: a "which-object" mask indicating what to pick and a "which-target-region" mask specifying where to place it. The resulting system processes RGB-D observations and natural-language task specifications to reactively generate multi-step pick-and-place actions for 3D box rearrangement. We conduct experiments across 11 task variants in warehouse-style environments with 1-30 boxes and diverse natural-language constraints. RAMP-3D achieves 79.5% success rate on long-horizon rearrangement tasks and significantly outperforms 2D VLM-based baselines, establishing mask-based reactive policies as a promising alternative to symbolic pipelines for long-horizon planning.

Zitieren

APA

Malik, A., Lowe, C., Shrestha, A., Lee, S., Li, F., & Fern, A. (2026). Grounding Vision and Language to 3D Masks for Long-Horizon Box Rearrangement. Proceedings of the International Conference on Automated Planning and Scheduling, 36(1), 729-737. https://doi.org/10.1609/icaps.v36i1.42893

Vancouver

Malik A, Lowe C, Shrestha A, Lee S, Li F, Fern A. Grounding Vision and Language to 3D Masks for Long-Horizon Box Rearrangement. Proceedings of the International Conference on Automated Planning and Scheduling. 2026;36(1):729-737. doi:10.1609/icaps.v36i1.42893

BibTeX
@article{Malik2026,
  author = {Malik, Ashish and Lowe, Caleb and Shrestha, Aayam and Lee, Stefan and Li, Fuxin and Fern, Alan},
  title = {Grounding Vision and Language to 3D Masks for Long-Horizon Box Rearrangement},
  journal = {Proceedings of the International Conference on Automated Planning and Scheduling},
  year = {2026},
  volume = {36},
  number = {1},
  pages = {729--737},
  doi = {10.1609/icaps.v36i1.42893}
}
Kostenlos starten →