EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Learning Where to Grasp and Place: Role-Specific Attention for Language-Conditioned Manipulation

Houyuan Xiao, Shengli Wang, Wenqing Zhang, Bin Wang, Xuancen Liu, Cheng Fu

Sensors · 18.11.2026 · 26 (22) · 7292

DOI: 10.3390/s26227292 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 18.11.2026. Verfasst von Houyuan Xiao und 5 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Language-conditioned tabletop manipulation requires identifying the object to be grasped and the spatial region for its placement relative to a reference object. This study presents an end-to-end model that predicts grasp position, grasp angle, and placement position jointly from an RGB image and a relational instruction. Role-specific attention maps weight a shared visual feature grid separately for grasping and placement, preserving spatial structure for action prediction. The maps are conditioned on the instruction through low-rank bilinear fusion, and the resulting model predicts operation parameters directly at inference. Existing scene-instruction pairs from the previous study are re-annotated with two position maps and grasp-angle classes to supervise the three output branches jointly. The model reaches full-pipeline success rates of 94.70% across 2000 simulated trials and 85.50% across 400 real-robot trials. In simulation and on the real robot, respectively, these rates exceed CLIPort-adapted by 4.15 and 4.25 percentage points and the modular baseline by 7.35 and 6.75 percentage points. Separate attention improves simulated full-pipeline success by 10.55 percentage points over shared attention; domain randomization improves real-robot success by 11.00 percentage points. Paraphrase and held-out-combination tests reveal remaining sensitivity to linguistic and compositional variation. These results indicate that role-specific attention supports joint learning of instruction-conditioned grasp and placement parameters in planar manipulation.

Zitieren

APA

Xiao, H., Wang, S., Zhang, W., Wang, B., Liu, X., & Fu, C. (2026). Learning Where to Grasp and Place: Role-Specific Attention for Language-Conditioned Manipulation. Sensors, 26(22), 7292. https://doi.org/10.3390/s26227292

Vancouver

Xiao H, Wang S, Zhang W, Wang B, Liu X, Fu C. Learning Where to Grasp and Place: Role-Specific Attention for Language-Conditioned Manipulation. Sensors. 2026;26(22):7292. doi:10.3390/s26227292

BibTeX
@article{Xiao2026,
  author = {Xiao, Houyuan and Wang, Shengli and Zhang, Wenqing and Wang, Bin and Liu, Xuancen and Fu, Cheng},
  title = {Learning Where to Grasp and Place: Role-Specific Attention for Language-Conditioned Manipulation},
  journal = {Sensors},
  year = {2026},
  volume = {26},
  number = {22},
  pages = {7292},
  doi = {10.3390/s26227292}
}
Kostenlos starten →