Learning Dexterous In-Hand Manipulation with Multifingered Hands via Visuomotor Diffusion

2025-03-04

AI Summary

This work addresses the vision–motor co-learning problem for in-hand manipulation tasks (e.g., unscrewing a bottle cap with a single multi-fingered dexterous hand). We propose an end-to-end vision–motor diffusion policy framework. Methodologically, we integrate AR-enhanced teleoperation to collect high-quality expert demonstrations; employ a joint HDBSCAN–GLOSH anomaly detection algorithm to automatically filter low-quality trajectories, significantly improving training robustness and cross-task generalization of the diffusion policy; and incorporate real-time inverse kinematics solving with motion retargeting for closed-loop control. The framework is successfully deployed on a physical Allegro four-fingered robotic hand, achieving high-precision, low-latency in-hand manipulation. All experimental videos and source code are publicly released.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

We present a framework for learning dexterous in-hand manipulation with multifingered hands using visuomotor diffusion policies. Our system enables complex in-hand manipulation tasks, such as unscrewing a bottle lid with one hand, by leveraging a fast and responsive teleoperation setup for the four-fingered Allegro Hand. We collect high-quality expert demonstrations using an augmented reality (AR) interface that tracks hand movements and applies inverse kinematics and motion retargeting for precise control. The AR headset provides real-time visualization, while gesture controls streamline teleoperation. To enhance policy learning, we introduce a novel demonstration outlier removal approach based on HDBSCAN clustering and the Global-Local Outlier Score from Hierarchies (GLOSH) algorithm, effectively filtering out low-quality demonstrations that could degrade performance. We evaluate our approach extensively in real-world settings and provide all experimental videos on the project website: https://dex-manip.github.io/

Problem

• Learning dexterous in-hand manipulation with multifingered hands.
• Enabling complex tasks like unscrewing bottle lids via teleoperation.
• Filtering low-quality demonstrations to enhance policy learning.

Innovation

• Visuomotor diffusion policies for dexterous manipulation
• AR interface with inverse kinematics for precise control
• HDBSCAN and GLOSH for demonstration outlier removal