Learning Physics-Guided Residual Dynamics for Deformable Object Simulation

2026-07-15

AI Summary

Accurately predicting the dynamics of deformable objects is crucial yet highly challenging for robotic manipulation. This work proposes the Physics-Guided Residual Dynamics (PGRD) framework, which integrates an optimizable mass-spring physical simulator with a neural network that corrects residual errors in the velocity domain, enabling stable and high-fidelity dynamics modeling. To capture long-range temporal dependencies, the method incorporates a sliding-window Transformer and represents system states using 3D Gaussian Splatting. Evaluated on diverse real-world deformable objects, PGRD significantly outperforms both purely physics-based and purely data-driven baselines. The approach demonstrates successful applications in model predictive control, language-guided manipulation, and interactive video prediction conditioned on action sequences.

Abstract

Simulating deformable objects is essential for a wide range of robotic manipulation applications, yet accurately predicting their dynamics remains challenging. We propose Physics-Guided Residual Dynamics (PGRD), a hybrid simulation framework that combines the advantages of physics-based and learning-based approaches. Specifically, PGRD combines an optimizable spring-mass simulator as a backbone with a learned neural network that predicts residual corrections to the physics-based predictions. We adopt a velocity-based formulation to ensure stable simulation and a sliding-window transformer architecture to capture temporal dependencies. We show that PGRD produces more accurate results than both purely physics-based and learning-based methods on a set of diverse real-world deformable objects. We further demonstrate the utility of PGRD in two applications: manipulation planning via Model Predictive Control, including a language-conditioned setting with a generated goal image; and interactive simulation via action-conditioned video prediction by 3D Gaussian Splatting.

Problem

• deformable object simulation
• dynamics prediction
• robotic manipulation
• physics-based simulation
• learning-based simulation

Innovation

• Physics-Guided Residual Dynamics
• hybrid simulation
• residual correction
• sliding-window transformer
• deformable object simulation