PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation

2025-04-23

AI Summary

This work addresses the poor robustness and limited generalization of policy learning for non-grasping manipulation tasks—such as pushing and poking—caused by high sensitivity to friction and coefficient of restitution. We propose an end-to-end differentiable physics-informed world model that jointly integrates differentiable physics simulation with Gaussian rasterization-based visual observation modeling. Without task-specific annotations, it identifies 3D rigid-body dynamics from merely five visual interaction trajectories. We introduce “physics-aware digital cousins”—a novel paradigm combining physics parameter randomization with model predictive control (MPC) to enable efficient Sim2Real transfer. Extensive evaluation in simulation and on real robotic platforms demonstrates high-precision pushing and poking control, strong cross-scenario generalization, and significant performance gains over state-of-the-art Real2Sim2Real approaches.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Low Level & Physics-based Vision
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

While non-prehensile manipulation (e.g., controlled pushing/poking) constitutes a foundational robotic skill, its learning remains challenging due to the high sensitivity to complex physical interactions involving friction and restitution. To achieve robust policy learning and generalization, we opt to learn a world model of the 3D rigid body dynamics involved in non-prehensile manipulations and use it for model-based reinforcement learning. We propose PIN-WM, a Physics-INformed World Model that enables efficient end-to-end identification of a 3D rigid body dynamical system from visual observations. Adopting differentiable physics simulation, PIN-WM can be learned with only few-shot and task-agnostic physical interaction trajectories. Further, PIN-WM is learned with observational loss induced by Gaussian Splatting without needing state estimation. To bridge Sim2Real gaps, we turn the learned PIN-WM into a group of Digital Cousins via physics-aware randomizations which perturb physics and rendering parameters to generate diverse and meaningful variations of the PIN-WM. Extensive evaluations on both simulation and real-world tests demonstrate that PIN-WM, enhanced with physics-aware digital cousins, facilitates learning robust non-prehensile manipulation skills with Sim2Real transfer, surpassing the Real2Sim2Real state-of-the-arts.

Problem

• Learning 3D rigid body dynamics for non-prehensile manipulation
• Enhancing Sim2Real transfer with physics-aware randomizations
• Robust policy learning without state estimation

Innovation

• Physics-INformed World Model for 3D dynamics
• Differentiable physics simulation enables few-shot learning
• Digital Cousins bridge Sim2Real via physics randomization