DextrAH-RGB: Visuomotor Policies to Grasp Anything with Dexterous Hands

2024-11-27

arXiv.org

AI Summary

Existing dexterous hand-arm coordinated grasping methods exhibit poor generalization in real-world scenarios, heavily relying on depth information and precise pose estimation. Method: This paper proposes the first end-to-end RGB-image-driven purely visual grasping framework. We introduce a privileged geometric fabric-guided policy (FGP) as a simulation-based teacher model and develop a lightweight RGB-only distillation framework to enable robust sim-to-real transfer. By integrating rasterization-based rendering, geometric fabric control, and reinforcement learning, our approach generates high-precision, contact-rich dynamic grasping motions without depth maps or explicit pose estimation. Results: The method achieves strong generalization across unseen geometries, textures, and lighting conditions, matching the performance of depth-based approaches while enabling real-time inference. It significantly enhances the practicality and deployment efficiency of dexterous manipulation in unstructured real-world environments.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

One of the most important, yet challenging, skills for a dexterous robot is grasping a diverse range of objects. Much of the prior work has been limited by speed, generality, or reliance on depth maps and object poses. In this paper, we introduce DextrAH-RGB, a system that can perform dexterous arm-hand grasping end-to-end from RGB image input. We train a privileged fabric-guided policy (FGP) in simulation through reinforcement learning that acts on a geometric fabric controller to dexterously grasp a wide variety of objects. We then distill this privileged FGP into a RGB-based FGP strictly in simulation using photorealistic tiled rendering. To our knowledge, this is the first work that is able to demonstrate robust sim2real transfer of an end2end RGB-based policy for complex, dynamic, contact-rich tasks such as dexterous grasping. DextrAH-RGB is competitive with depth-based dexterous grasping policies, and generalizes to novel objects with unseen geometry, texture, and lighting conditions in the real world. Videos of our system grasping a diverse range of unseen objects are available at url{https://dextrah-rgb.github.io/}.

Problem

• Robot Manipulation
• Adaptive Grasping
• Real-world Conditions

Innovation

• DextrAH-RGB
• RGB-based learning
• Virtual to real-world adaptation