Object-Mesh-Free Dexterous Robot Demonstrations from Human RGB-D Video and Teleoperation
Machines · 05.11.2026 · 14 (11) · 1303
DOI: 10.3390/machines14111303 ↗Kurz gefasst
Fachartikel in der Zeitschrift Machines, online erschienen am 05.11.2026. Verfasst von Juyong Hong und 9 weiteren Autor:innen. Die Arbeit nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Trade, Industry and Energy (MOTIE), Republic of Korea.
Abstract (Originalsprache)
Teleoperating a five-fingered robot hand to collect demonstrations is costly and scales poorly, while human video is abundant yet contains neither robot commands nor robot images. Existing approaches overlay rendered robots on human video, mainly for parallel-jaw grippers; reconstruct object meshes or point clouds in place of the original RGB; or require a wearable exoskeleton during capture. We present a pipeline that composes stationary RGB-D video of a bare human hand with teleoperation logs of the same task. The video supplies the metric hand trajectory and the scene; the teleoperation log supplies recorded finger position targets. Persistent hand-state transition alignment (PHTA) matches ordered changes in a common hand-shape representation and transfers the complete finger-command vector through one temporal correspondence, without task-phase, contact, or grasp labels. Part-aware visibility transfer replaces the human hand with the robot hand while preserving which parts were in front of or behind the object, requiring no object mesh or pose. Combining N videos with M demonstrations yields N×M composed demonstrations that reuse the M recorded finger-command trajectories across the N scenes. On four real-robot tasks, five teleoperation demonstrations composed with twenty human episodes raise behavior-cloning success from 0–45% to 50–90%, reaching in two tasks the same observed success rate as in one hundred teleoperation demonstrations.
Förderung
- Ministry of Trade, Industry and Energy (MOTIE), Republic of Korea
Zitieren
APA
Hong, J., Um, S., Son, Y. G., Kwon, C. H., Ahn, C., Bae, T., Leeu, S., Yun, T., Ko, T., & Choi, H. R. (2026). Object-Mesh-Free Dexterous Robot Demonstrations from Human RGB-D Video and Teleoperation. Machines, 14(11), 1303. https://doi.org/10.3390/machines14111303
Vancouver
Hong J, Um S, Son YG, Kwon CH, Ahn C, Bae T, et al. Object-Mesh-Free Dexterous Robot Demonstrations from Human RGB-D Video and Teleoperation. Machines. 2026;14(11):1303. doi:10.3390/machines14111303
BibTeX
@article{Hong2026,
author = {Hong, Juyong and Um, Seunghwan and Son, Young Gwang and Kwon, Chul Hyun and Ahn, Chemin and Bae, Taehyun and Leeu, Sihoon and Yun, Taehwan and Ko, Taiho and Choi, Hyouk Ryeol},
title = {Object-Mesh-Free Dexterous Robot Demonstrations from Human RGB-D Video and Teleoperation},
journal = {Machines},
year = {2026},
volume = {14},
number = {11},
pages = {1303},
doi = {10.3390/machines14111303}
}
EUVIMED