OrbiSim: World Models as Differentiable Physics Engines for Embodied Intelligence

2026-05-12

AI Summary

This work addresses the limitations of traditional world models, which often lack physical consistency and differentiability, thereby hindering gradient-based policy optimization and precise physical reasoning. The authors propose a fully differentiable, physics engine–inspired world model that establishes an end-to-end differentiable pathway from state transitions to visual observations by integrating structured scene representations, neural dynamics modeling, and differentiable rendering. This model is the first to unify physically plausible simulation with full differentiability, enabling differentiable contact modeling and effective policy gradient optimization even under sparse rewards. Experimental results demonstrate that the proposed approach significantly outperforms existing methods in prediction accuracy, control performance, and sensitivity to changes in asset configuration and physical parameters.

Abstract

We present OrbiSim, a novel robotic simulation paradigm that redefines world models as a fully differentiable physics engine for embodied intelligence. Unlike prior world models that focus on unconstrained imagination in latent or visual domains, OrbiSim establishes a unified, physically-grounded pathway that bridges structured scene assets, neural dynamics, and downstream reinforcement learning. By enabling end-to-end differentiability throughout the entire simulation loop -- spanning from explicit state transitions to visual observation generation -- OrbiSim supports tasks traditionally intractable for classical simulators, such as differentiable contact modeling, gradient-based policy optimization under sparse rewards, and intuitive physical inference. Empirical results demonstrate that OrbiSim significantly outperforms state-of-the-art world models in both predictive fidelity and control performance. Furthermore, its consistent responsiveness to asset configurations and physical parameters suggests its potential as a differentiable tool for enhancing robot simulation and policy training.

Problem

• world models
• differentiable physics
• embodied intelligence
• robot simulation
• gradient-based policy optimization

Innovation

• differentiable physics engine
• world models
• embodied intelligence
• end-to-end differentiability
• gradient-based policy optimization