Learning How the World Evolves: Extrapolative Video World Models via Latent Dynamics Reasoning

2026-08-10

AI Summary

Current video diffusion models solely fit pixel-level data while neglecting real-world physical dynamics, leading to distorted predictions in out-of-distribution scenarios. This work proposes Latent Dynamics Reasoning (LDR), which models state transitions in a structured latent space as explicit kinematic integration, learning only residual dynamics of third order and higher. LDR establishes the first extrapolatable video world model, achieving over a 20-fold reduction in the gap between in- and out-of-distribution prediction errors compared to diffusion baselines at 256² resolution. It further reduces model parameters by 26× and accelerates inference by 143×, while generalizing to unseen objects, motion directions, and attributes—including accurate prediction of reversed motion trajectories.

Abstract

The world evolves following its dynamics, i.e., its laws of motion. However, leading video diffusion models largely fit the pixels without modeling how the pixels transit over time. Thus, they render visually plausible frames but may not accurately obey the laws. To capture the dynamics purely from pixels, we introduce Latent Dynamics Reasoning (LDR). LDR casts the latent transition as an explicit kinematic integration, where the lower-order dynamics are integrated numerically and the model regresses only the third- and higher-order residual that drives the rollout. For this integration to extrapolate better, LDR runs it on a structured latent rather than dense convolutional features. Following PhyWorld, we validate LDR on a controlled white-box physics benchmark spanning five tasks (uniform motion, parabola, collision, bouncing, looming), focusing on out-of-distribution scenarios that reveal whether a model has truly learned the underlying dynamics. LDR extrapolates the learned dynamics far better: the gap between its in- and out-of-distribution error is over 20$\times$ smaller than the video diffusion baseline's, under both single- and joint-task training at 256$^2$ resolution, while using 26$\times$ fewer parameters and running 143$\times$ faster. LDR can even generalize under severe shift: for example, trained only on red balls moving left-to-right, it correctly predicts the motion of a blue square moving right-to-left. To our knowledge, this is the first video world model that extrapolates learned dynamics beyond its training distribution. Project page: https://lat-dyn-reason.github.io/

Problem

• video world models
• dynamics modeling
• extrapolation
• latent dynamics
• out-of-distribution generalization

Innovation

• Latent Dynamics Reasoning
• video world models
• dynamics extrapolation
• structured latent space
• physics-based video prediction