DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Model

2025-10-09

AI Summary

This work addresses the sim-to-real transfer challenge in dexterous hand manipulation—specifically, generalizing in-hand object rotation under realistic conditions involving complex geometries, high aspect ratios (up to 5.33), small-scale objects, and arbitrary wrist orientations. We propose a joint-level neuromechanical transfer framework that decouples joint dynamics modeling, implicitly encodes system coupling, compresses state representation into low-dimensional latent variables, and integrates simulation-policy adaptation with fully automated, unsupervised real-world data collection. Leveraging only minimal real-world interaction data, a single learned policy robustly manipulates diverse extreme objects—including animal figurines and slender parts—without task-specific retraining. Experimental validation on physical hardware demonstrates strong cross-object and cross-pose generalization, confirming practical efficacy in real-world settings.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Achieving generalized in-hand object rotation remains a significant challenge in robotics, largely due to the difficulty of transferring policies from simulation to the real world. The complex, contact-rich dynamics of dexterous manipulation create a"reality gap"that has limited prior work to constrained scenarios involving simple geometries, limited object sizes and aspect ratios, constrained wrist poses, or customized hands. We address this sim-to-real challenge with a novel framework that enables a single policy, trained in simulation, to generalize to a wide variety of objects and conditions in the real world. The core of our method is a joint-wise dynamics model that learns to bridge the reality gap by effectively fitting limited amount of real-world collected data and then adapting the sim policy's actions accordingly. The model is highly data-efficient and generalizable across different whole-hand interaction distributions by factorizing dynamics across joints, compressing system-wide influences into low-dimensional variables, and learning each joint's evolution from its own dynamic profile, implicitly capturing these net effects. We pair this with a fully autonomous data collection strategy that gathers diverse, real-world interaction data with minimal human intervention. Our complete pipeline demonstrates unprecedented generality: a single policy successfully rotates challenging objects with complex shapes (e.g., animals), high aspect ratios (up to 5.33), and small sizes, all while handling diverse wrist orientations and rotation axes. Comprehensive real-world evaluations and a teleoperation application for complex tasks validate the effectiveness and robustness of our approach. Website: https://meowuu7.github.io/DexNDM/

Problem

• Bridging sim-to-real gap for dexterous in-hand object rotation
• Generalizing manipulation policies across diverse objects and conditions
• Overcoming constrained scenarios with complex contact-rich dynamics

Innovation

• Joint-wise neural dynamics model bridges reality gap
• Data-efficient factorization learns dynamics across joints
• Autonomous data collection enables diverse real-world generalization