Learning Granularity-Aware Affordances from Human-Object Interaction for Tool-Based Functional Grasping in Dexterous Robotics

2024-06-30

arXiv.org

AI Summary

To address the challenge of functional grasping in dexterous robotic tool manipulation, this paper proposes GAAF-Dex, an end-to-end framework that learns granularity-aware affordance features from human-object interactions to jointly localize functional contact regions (fine-grained) and predict dexterous grasp poses (coarse-grained). We introduce a novel weakly supervised cross-view learning paradigm—leveraging exocentric supervision to guide egocentric affordance estimation—and a force-feedback-driven coarse-to-fine grasp post-processing module. The method integrates multi-granularity affordance representation, functional finger coordinate localization, hand-to-end-effector coordinate transformation, and joint modeling of exocentric and egocentric images. Evaluated on our newly constructed FAH dataset (6K images, 18 tools, 6 task categories), GAAF-Dex significantly outperforms state-of-the-art methods in both functional region localization accuracy and dexterous gesture prediction. The code is publicly available.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Biometrics, Face, Gesture & Pose
• Humans and AI: Interaction Techniques and Devices

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

To enable robots to use tools, the initial step is teaching robots to employ dexterous gestures for touching specific areas precisely where tasks are performed. Affordance features of objects serve as a bridge in the functional interaction between agents and objects. However, leveraging these affordance cues to help robots achieve functional tool grasping remains unresolved. To address this, we propose a granularity-aware affordance feature extraction method for locating functional affordance areas and predicting dexterous coarse gestures. We study the intrinsic mechanisms of human tool use. On one hand, we use fine-grained affordance features of object-functional finger contact areas to locate functional affordance regions. On the other hand, we use highly activated coarse-grained affordance features in hand-object interaction regions to predict grasp gestures. Additionally, we introduce a model-based post-processing module that includes functional finger coordinate localization, finger-to-end coordinate transformation, and force feedback-based coarse-to-fine grasping. This forms a complete dexterous robotic functional grasping framework GAAF-Dex, which learns Granularity-Aware Affordances from human-object interaction for tool-based Functional grasping in Dexterous Robotics. Unlike fully-supervised methods that require extensive data annotation, we employ a weakly supervised approach to extract relevant cues from exocentric (Exo) images of hand-object interactions to supervise feature extraction in egocentric (Ego) images. We have constructed a small-scale dataset, FAH, which includes near 6K images of functional hand-object interaction Exo- and Ego images of 18 commonly used tools performing 6 tasks. Extensive experiments on the dataset demonstrate our method outperforms state-of-the-art methods. The code will be made publicly available at https://github.com/yangfan293/GAAF-DEX.

Problem

• Teaching robots dexterous tool grasping using affordance features
• Locating functional affordance areas via granularity-aware extraction
• Predicting grasp gestures from hand-object interaction cues

Innovation

• Granularity-aware affordance feature extraction method
• Weakly supervised learning from Exo and Ego images
• Model-based post-processing for robotic action execution