D(R, O) Grasp: A Unified Representation of Robot and Object Interaction for Cross-Embodiment Dexterous Grasping

2024-10-02

arXiv.org

AI Summary

To address the challenge of generalizing dexterous grasping across diverse robotic hands and arbitrary object geometries, this paper proposes the D(R,O) Grasp framework. The method introduces a novel hand-object joint implicit representation, unifying the modeling of hand configurations and object point clouds to enable zero-shot transfer to unseen hand morphologies and objects. Implemented as an end-to-end neural network, it integrates a point cloud encoder with a hand-object interaction decoder, jointly optimizing for kinematic feasibility, force-closure stability, and grasp diversity. In simulation, the framework achieves an average success rate of 87.53% (inference time <1 s per grasp); on the real-world LeapHand platform, it attains 89%, significantly outperforming prior approaches. Its core contribution lies in the first formulation of geometric decoupling between hand and object representations, enabling cross-morphology zero-shot grasping generalization.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Dexterous grasping is a fundamental yet challenging skill in robotic manipulation, requiring precise interaction between robotic hands and objects. In this paper, we present $mathcal{D(R,O)}$ Grasp, a novel framework that models the interaction between the robotic hand in its grasping pose and the object, enabling broad generalization across various robot hands and object geometries. Our model takes the robot hand's description and object point cloud as inputs and efficiently predicts kinematically valid and stable grasps, demonstrating strong adaptability to diverse robot embodiments and object geometries. Extensive experiments conducted in both simulated and real-world environments validate the effectiveness of our approach, with significant improvements in success rate, grasp diversity, and inference speed across multiple robotic hands. Our method achieves an average success rate of 87.53% in simulation in less than one second, tested across three different dexterous robotic hands. In real-world experiments using the LeapHand, the method also demonstrates an average success rate of 89%. $mathcal{D(R,O)}$ Grasp provides a robust solution for dexterous grasping in complex and varied environments. The code, appendix, and videos are available on our project website at https://nus-lins-lab.github.io/drograspweb/.

Problem

• Modeling robot-object interaction for dexterous grasping.
• Generalizing grasp prediction across diverse robot hands and objects.
• Improving grasp success rate, diversity, and inference speed.

Innovation

• Unified framework for robot-object interaction modeling
• Inputs: robot hand description, object point cloud
• Efficient prediction of stable, kinematically valid grasps