Learning Where to Grasp and Place: Role-Specific Attention for Language-Conditioned Manipulation
Sensors · 18.11.2026 · 26 (22) · 7292
DOI: 10.3390/s26227292 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 18.11.2026. Verfasst von Houyuan Xiao und 5 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Language-conditioned tabletop manipulation requires identifying the object to be grasped and the spatial region for its placement relative to a reference object. This study presents an end-to-end model that predicts grasp position, grasp angle, and placement position jointly from an RGB image and a relational instruction. Role-specific attention maps weight a shared visual feature grid separately for grasping and placement, preserving spatial structure for action prediction. The maps are conditioned on the instruction through low-rank bilinear fusion, and the resulting model predicts operation parameters directly at inference. Existing scene-instruction pairs from the previous study are re-annotated with two position maps and grasp-angle classes to supervise the three output branches jointly. The model reaches full-pipeline success rates of 94.70% across 2000 simulated trials and 85.50% across 400 real-robot trials. In simulation and on the real robot, respectively, these rates exceed CLIPort-adapted by 4.15 and 4.25 percentage points and the modular baseline by 7.35 and 6.75 percentage points. Separate attention improves simulated full-pipeline success by 10.55 percentage points over shared attention; domain randomization improves real-robot success by 11.00 percentage points. Paraphrase and held-out-combination tests reveal remaining sensitivity to linguistic and compositional variation. These results indicate that role-specific attention supports joint learning of instruction-conditioned grasp and placement parameters in planar manipulation.
Zitieren
APA
Xiao, H., Wang, S., Zhang, W., Wang, B., Liu, X., & Fu, C. (2026). Learning Where to Grasp and Place: Role-Specific Attention for Language-Conditioned Manipulation. Sensors, 26(22), 7292. https://doi.org/10.3390/s26227292
Vancouver
Xiao H, Wang S, Zhang W, Wang B, Liu X, Fu C. Learning Where to Grasp and Place: Role-Specific Attention for Language-Conditioned Manipulation. Sensors. 2026;26(22):7292. doi:10.3390/s26227292
BibTeX
@article{Xiao2026,
author = {Xiao, Houyuan and Wang, Shengli and Zhang, Wenqing and Wang, Bin and Liu, Xuancen and Fu, Cheng},
title = {Learning Where to Grasp and Place: Role-Specific Attention for Language-Conditioned Manipulation},
journal = {Sensors},
year = {2026},
volume = {26},
number = {22},
pages = {7292},
doi = {10.3390/s26227292}
}
EUVIMED