Retrieval Dexterity: Efficient Object Retrieval in Clutters with Dexterous Hand

2025-02-25

AI Summary

In cluttered multi-object stacking scenarios, occlusion of target objects severely degrades retrieval efficiency and necessitates sequential grasping—a key bottleneck. To address this, we propose a dexterous hand-arm coordinated retrieval method leveraging non-grasping obstacle-clearing actions (pushing, stirring, poking). Our approach introduces a large-scale parallel reinforcement learning framework based on Proximal Policy Optimization (PPO), enabling autonomous evolution of generalizable clearing policies within a multimodal simulation environment. Domain-intrinsic adaptation bridges the sim-to-real gap, ensuring efficient transfer to physical multifinger dexterous hands. Crucially, the method eliminates reliance on high-precision grasping. Evaluated on over ten household objects under diverse clutter configurations, it significantly improves both retrieval success rate and speed while maintaining real-time performance and robustness. The system has been successfully deployed on a physical robotic platform.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Retrieving objects buried beneath multiple objects is not only challenging but also time-consuming. Performing manipulation in such environments presents significant difficulty due to complex contact relationships. Existing methods typically address this task by sequentially grasping and removing each occluding object, resulting in lengthy execution times and requiring impractical grasping capabilities for every occluding object. In this paper, we present a dexterous arm-hand system for efficient object retrieval in multi-object stacked environments. Our approach leverages large-scale parallel reinforcement learning within diverse and carefully designed cluttered environments to train policies. These policies demonstrate emergent manipulation skills (e.g., pushing, stirring, and poking) that efficiently clear occluding objects to expose sufficient surface area of the target object. We conduct extensive evaluations across a set of over 10 household objects in diverse clutter configurations, demonstrating superior retrieval performance and efficiency for both trained and unseen objects. Furthermore, we successfully transfer the learned policies to a real-world dexterous multi-fingered robot system, validating their practical applicability in real-world scenarios. Videos can be found on our project website https://ChangWinde.github.io/RetrDex.

Problem

• Efficient object retrieval in clutter
• Dexterous hand manipulation skills
• Real-world robotic system application

Innovation

• Dexterous arm-hand system
• Large-scale parallel reinforcement learning
• Emergent manipulation skills