D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Grasping

2026-03-01

AI Summary

This work addresses the sim-to-real transfer challenge in dexterous grasping caused by mismatches in dynamic parameters such as object mass. The authors propose a differentiable real-to-sim-to-real engine that integrates Gaussian splatting representations, differentiable physics simulation, and reinforcement learning to jointly optimize object mass from real-world visual observations and robot control signals, thereby constructing high-fidelity digital twins. Additionally, they introduce a novel force-aware grasping policy transfer mechanism that leverages only a few human demonstrations to efficiently generate simulation training data. Experiments demonstrate that the method achieves high-accuracy mass estimation across diverse geometric and inertial conditions, significantly improving grasp success rates and effectively bridging the performance gap between simulation and reality.

Technology Category

• Intelligent Robots: Manipulation
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Web Mining and Content Analysis: Web data generation and simulation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Simulation provides a cost-effective and flexible platform for data generation and policy learning to develop robotic systems. However, bridging the gap between simulation and real-world dynamics remains a significant challenge, especially in physical parameter identification. In this work, we introduce a real-to-sim-to-real engine that leverages the Gaussian Splat representations to build a differentiable engine, enabling object mass identification from real-world visual observations and robot control signals, while enabling grasping policy learning simultaneously. Through optimizing the mass of the manipulated object, our method automatically builds high-fidelity and physically plausible digital twins. Additionally, we propose a novel approach to train force-aware grasping policies from limited data by transferring feasible human demonstrations into simulated robot demonstrations. Through comprehensive experiments, we demonstrate that our engine achieves accurate and robust performance in mass identification across various object geometries and mass values. Those optimized mass values facilitate force-aware policy learning, achieving superior and high performance in object grasping, effectively reducing the sim-to-real gap.

Problem

• sim-to-real gap
• physical parameter identification
• dexterous grasping
• digital twin
• robotic manipulation

Innovation

• Differentiable Simulation
• Gaussian Splatting
• Digital Twin
• Force-aware Grasping
• Sim-to-Real Transfer