Learning Dexterous Manipulation Skills from Imperfect Simulations

2025-12-01

AI Summary

Addressing the challenge of high-fidelity simulation of haptic feedback and complex contact dynamics, this paper proposes a three-stage sim-to-real framework. First, dexterous manipulation policies are pre-trained in a simplified simulator. Second, multimodal demonstrations—including rich haptic signals—are collected via real-world teleoperation and integrated into the policy via behavior cloning to inject physical realism. Third, reinforcement learning fine-tunes the policy to enhance robustness. This approach significantly mitigates simulation-to-reality mismatch, enabling efficient zero-shot transfer to unseen object geometries and external disturbances. Experiments demonstrate superior task completion rates over direct sim-to-real baselines—particularly in nut assembly and screw tightening—while maintaining strong generalization across diverse object shapes and dynamic perturbations. The framework establishes a scalable, haptics-aware paradigm for dexterous robotic manipulation, advancing the state of sim-to-real transfer in contact-rich tasks.

Technology Category

• Intelligent Robots: Manipulation
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reinforcement learning and sim-to-real transfer have made significant progress in dexterous manipulation. However, progress remains limited by the difficulty of simulating complex contact dynamics and multisensory signals, especially tactile feedback. In this work, we propose ours, a sim-to-real framework that addresses these limitations and demonstrates its effectiveness on nut-bolt fastening and screwdriving with multi-fingered hands. The framework has three stages. First, we train reinforcement learning policies in simulation using simplified object models that lead to the emergence of correct finger gaits. We then use the learned policy as a skill primitive within a teleoperation system to collect real-world demonstrations that contain tactile and proprioceptive information. Finally, we train a behavior cloning policy that incorporates tactile sensing and show that it generalizes to nuts and screwdrivers with diverse geometries. Experiments across both tasks show high task progress ratios compared to direct sim-to-real transfer and robust performance even on unseen object shapes and under external perturbations. Videos and code are available on https://dexscrew.github.io.

Problem

• Simulating complex contact dynamics and tactile feedback
• Transferring dexterous manipulation skills from simulation to reality
• Generalizing learned skills to diverse object geometries and perturbations

Innovation

• Sim-to-real transfer with simplified object models
• Teleoperation for real-world tactile demonstration collection
• Behavior cloning incorporating tactile sensing generalization