Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids

2025-02-27

AI Summary

This work addresses key Sim-to-Real transfer challenges in dexterous manipulation for humanoid robots—namely, contact dynamics mismatch, visual perception bias, and sparse rewards. We propose four core technical contributions: (1) an automatic real-to-sim tuning module that dynamically calibrates simulation parameters; (2) a generalizable long-horizon reward function to mitigate reward sparsity; (3) a divide-and-conquer policy distillation framework that decouples motion control from contact decision-making; and (4) a hybrid object representation integrating geometric and material priors. Our approach enables end-to-end vision-driven control without human demonstrations. Evaluated on three dexterous manipulation tasks, it achieves high-performance zero-shot Sim-to-Real transfer, significantly improving sample efficiency and cross-domain generalization. Ablation studies confirm the effectiveness of each component.

Technology Category

• Intelligent Robots: Manipulation
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Reinforcement learning has delivered promising results in achieving human- or even superhuman-level capabilities across diverse problem domains, but success in dexterous robot manipulation remains limited. This work investigates the key challenges in applying reinforcement learning to solve a collection of contact-rich manipulation tasks on a humanoid embodiment. We introduce novel techniques to overcome the identified challenges with empirical validation. Our main contributions include an automated real-to-sim tuning module that brings the simulated environment closer to the real world, a generalized reward design scheme that simplifies reward engineering for long-horizon contact-rich manipulation tasks, a divide-and-conquer distillation process that improves the sample efficiency of hard-exploration problems while maintaining sim-to-real performance, and a mixture of sparse and dense object representations to bridge the sim-to-real perception gap. We show promising results on three humanoid dexterous manipulation tasks, with ablation studies on each technique. Our work presents a successful approach to learning humanoid dexterous manipulation using sim-to-real reinforcement learning, achieving robust generalization and high performance without the need for human demonstration.

Problem

• Sim-to-real learning for dexterous robot manipulation
• Overcoming challenges in contact-rich humanoid tasks
• Generalized reward design for long-horizon manipulation

Innovation

• Automated real-to-sim tuning module
• Generalized reward design scheme
• Divide-and-conquer distillation process