EUVIMEDEUVIMED entdecken
Crossref

A retrieval-augmented framework enabling VLM spatial awareness for object-centric robot manipulation

Kai Chen iD, Chengkun Li, Chang Tu, Jiahui Pan, Yiyao Ma, Wei Chen, Zhongxiang Zhou iD, Xuecheng Xu iD, Stephen James, Chi-Wing Fu, Rong Xiong iD, Pieter Abbeel iD und 2 weitere

Science Robotics · 29.04.2026 · 11 (113)

DOI: 10.1126/scirobotics.aea2092 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Science Robotics, erschienen am 29.04.2026. Verfasst von Kai Chen und 13 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Engineering, Chinese University of Hong Kong, HKSAR, China. Die Arbeit nennt 60 Literaturangaben. Gefördert von: Hong Kong Centre for Logistics Robotics, National Natural Science Foundation of China, Research Grants Council, University Grants Committee, Hong Kong.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Connecting the semantic reasoning of vision-language models (VLMs) to the precise geometric demands of robotic manipulation remains a fundamental challenge. Although VLMs can interpret high-level commands, they lack the intrinsic spatial intelligence required for tasks demanding precise object placement, orientation, and physical reasoning. Here, we introduce Retrieval-Augmented Manipulation (RAM), an object-centric framework that endows general-purpose vision foundation models with the spatial reasoning necessary for robust manipulation. RAM bridges the semantic-to-geometric gap by grounding abstract concepts into an explicit, object-centric three-dimensional (3D) representation. This grounded information is then provided as augmented context to the VLM, empowering it to decompose complex instructions into a sequence of spatially precise and physically plausible subgoals. We demonstrate that RAM, in a zero-shot setting on a real-world robot, can execute these subgoals to fulfill complex spatial language instructions, complete spatially aware manipulation under the guidance of a single 2D image, and adaptively replan tasks by reasoning about physical constraints like object size and collisions. Quantitative evaluations on the Common Object in 3D (CO3D) dataset also validated that RAM’s core vision module generalizes to previously unseen object categories and is robust to variations in shape and occlusions. By providing a structured bridge between semantic intent and geometric execution, RAM represents a critical step toward developing more physically intelligent and general-purpose robotic systems.

Autor:innen und Institutionen

  • Department of Computer Science and Engineering, Chinese University of Hong Kong, HKSAR, China.
  • Department of Mechanical and Automation Engineering, Chinese University of Hong Kong, HKSAR, China.
  • Zhejiang Humanoid Robot Innovation Center Co. Ltd., Ningbo, Zhejiang, China.
  • Imperial College London, London, UK.
  • College of Control Science and Engineering, Zhejiang University, Hangzhou, Zhejiang, China.
  • University of California, Berkeley, CA, USA.

ORCID-Profile: Kai Chen, Zhongxiang Zhou, Xuecheng Xu, Rong Xiong, Pieter Abbeel, Yun-Hui Liu, Qi Dou

Förderung

  • Hong Kong Centre for Logistics Robotics
  • National Natural Science Foundation of China
  • Research Grants Council, University Grants Committee, Hong Kong
  • Joint Funds of the National Natural Science Foundation of China

Zitieren

APA

Chen, K., Li, C., Tu, C., Pan, J., Ma, Y., Chen, W., Zhou, Z., Xu, X., James, S., Fu, C. W., Xiong, R., Abbeel, P., Liu, Y. H., & Dou, Q. (2026). A retrieval-augmented framework enabling VLM spatial awareness for object-centric robot manipulation. Science Robotics, 11(113). https://doi.org/10.1126/scirobotics.aea2092

Vancouver

Chen K, Li C, Tu C, Pan J, Ma Y, Chen W, et al. A retrieval-augmented framework enabling VLM spatial awareness for object-centric robot manipulation. Science Robotics. 2026;11(113). doi:10.1126/scirobotics.aea2092

BibTeX
@article{Chen2026,
  author = {Chen, Kai and Li, Chengkun and Tu, Chang and Pan, Jiahui and Ma, Yiyao and Chen, Wei and Zhou, Zhongxiang and Xu, Xuecheng and James, Stephen and Fu, Chi-Wing and Xiong, Rong and Abbeel, Pieter and Liu, Yun-Hui and Dou, Qi},
  title = {A retrieval-augmented framework enabling VLM spatial awareness for object-centric robot manipulation},
  journal = {Science Robotics},
  year = {2026},
  volume = {11},
  number = {113},
  doi = {10.1126/scirobotics.aea2092}
}
Kostenlos starten →