NeuralFeels with neural fields: Visuotactile perception for in-hand manipulation
Science Robotics · 2024
DOI: 10.1126/scirobotics.adl0628 ↗Kurz gefasst
Fachartikel in der Zeitschrift Science Robotics, 2024. Verfasst von Sudharshan Suresh und 11 weiteren Autor:innen. Beteiligte Einrichtung: Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA. Hauptthemen laut MEDLINE: Hand, Robotics, Visual Perception, Touch Perception. Die Arbeit wurde bisher 103-mal zitiert (Crossref) und nennt 115 Literaturangaben.
Abstract (Originalsprache)
To achieve human-level dexterity, robots must infer spatial awareness from multimodal sensing to reason over contact interactions. During in-hand manipulation of novel objects, such spatial awareness involves estimating the object’s pose and shape. The status quo for in-hand perception primarily uses vision and is restricted to tracking a priori known objects. Moreover, visual occlusion of objects in hand is imminent during manipulation, preventing current systems from pushing beyond tasks without occlusion. We combined vision and touch sensing on a multifingered hand to estimate an object’s pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. We studied multimodal in-hand perception in simulation and the real world, interacting with different objects via a proprioception-driven policy. Our experiments showed final reconstruction F scores of 81% and average pose drifts of 4.7 millimeters, which was further reduced to 2.3 millimeters with known object models. In addition, we observed that, under heavy visual occlusion, we could achieve improvements in tracking up to 94% compared with vision-only methods. Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation. We release our evaluation dataset of 70 experiments, FeelSight, as a step toward benchmarking in this domain. Our neural representation driven by multimodal sensing can serve as a perception backbone toward advancing robot dexterity.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA.
- FAIR, Meta, Menlo Park, CA 94025, USA.
- Department of Electrical Engineering and Computer Sciences, UC Berkeley, Berkeley, CA 94720, USA.
- Institute of Artificial Intelligence, Technische Universität Dresden, 01062 Dresden, Germany.
- Centre for Tactile Internet with Human-in-the-Loop (CeTI), 01062 Dresden, Germany.
ORCID-Profile: Sudharshan Suresh, Taosha Fan, Jitendra Malik, Mrinal Kalakrishnan, Roberto Calandra, Michael Kaess
Zitieren
APA
Suresh, S., Qi, H., Wu, T., Fan, T., Pineda, L., Lambeta, M., Malik, J., Kalakrishnan, M., Calandra, R., Kaess, M., Ortiz, J., & Mukadam, M. (2024). NeuralFeels with neural fields: Visuotactile perception for in-hand manipulation. Science Robotics. https://doi.org/10.1126/scirobotics.adl0628
Vancouver
Suresh S, Qi H, Wu T, Fan T, Pineda L, Lambeta M, et al. NeuralFeels with neural fields: Visuotactile perception for in-hand manipulation. Science Robotics. 2024. doi:10.1126/scirobotics.adl0628
BibTeX
@article{Suresh2024,
author = {Suresh, Sudharshan and Qi, Haozhi and Wu, Tingfan and Fan, Taosha and Pineda, Luis and Lambeta, Mike and Malik, Jitendra and Kalakrishnan, Mrinal and Calandra, Roberto and Kaess, Michael and Ortiz, Joseph and Mukadam, Mustafa},
title = {NeuralFeels with neural fields: Visuotactile perception for in-hand manipulation},
journal = {Science Robotics},
year = {2024},
doi = {10.1126/scirobotics.adl0628}
}
EUVIMED