EUVIMEDEUVIMED entdecken
Crossref

Learning dexterity from human hand motion in internet videos

Kenneth Shaw, Shikhar Bahl, Aravind Sivakumar, Aditya Kannan, Deepak Pathak

The International Journal of Robotics Research · 2024-04 · 43 (4) · 513-532

DOI: 10.1177/02783649241227559 ↗

Kurz gefasst

Fachartikel in der Zeitschrift The International Journal of Robotics Research, online erschienen am 22.01.2024. Verfasst von Kenneth Shaw und 4 weiteren Autor:innen. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 84 Literaturangaben. Gefördert von: Samsung GRO, ONR, NSF.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

To build general robotic agents that can operate in many environments, it is often useful for robots to collect experience in the real world. However, unguided experience collection is often not feasible due to safety, time, and hardware restrictions. We thus propose leveraging the next best thing as real world experience: videos of humans using their hands. To utilize these videos, we develop a method that retargets any 1st person or 3rd person video of human hands and arms into the robot hand and arm trajectories. While retargeting is a difficult problem, our key insight is to rely on only internet human hand video to train it. We use this method to present results in two areas: First, we build a system that enables any human to control a robot hand and arm, simply by demonstrating motions with their own hand. The robot observes the human operator via a single RGB camera and imitates their actions in real-time . This enables the robot to collect real-world experience safely using supervision. See these results at https://robotic-telekinesis.github.io . Second, we retarget in-the-wild human internet video into task-conditioned pseudo-robot trajectories to use as artificial robot experience. This learning algorithm leverages action priors from human hand actions, visual features from the images, and physical priors from dynamical systems to pretrain typical human behavior for a particular robot task. We show that by leveraging internet human hand experience, we need fewer robot demonstrations compared to many other methods. See these results at https://video-dex.github.io

Förderung

  • Samsung GRO
  • ONR
  • NSF
  • NSF GRFP
  • GoodAI Research Award

Zitieren

APA

Shaw, K., Bahl, S., Sivakumar, A., Kannan, A., & Pathak, D. (2024). Learning dexterity from human hand motion in internet videos. The International Journal of Robotics Research, 43(4), 513-532. https://doi.org/10.1177/02783649241227559

Vancouver

Shaw K, Bahl S, Sivakumar A, Kannan A, Pathak D. Learning dexterity from human hand motion in internet videos. The International Journal of Robotics Research. 2024;43(4):513-532. doi:10.1177/02783649241227559

BibTeX
@article{Shaw2024,
  author = {Shaw, Kenneth and Bahl, Shikhar and Sivakumar, Aravind and Kannan, Aditya and Pathak, Deepak},
  title = {Learning dexterity from human hand motion in internet videos},
  journal = {The International Journal of Robotics Research},
  year = {2024},
  volume = {43},
  number = {4},
  pages = {513--532},
  doi = {10.1177/02783649241227559}
}
Kostenlos starten →