ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models

2026-05-08

AI Summary

Existing action-conditioned world models lack systematic evaluation on diverse physical interactions, hindering their capacity for general physical understanding. This work proposes ACWM-Phys—the first controllable action-conditioned video prediction benchmark encompassing a broad spectrum of physical dynamics, including rigid bodies, deformable objects, and particle systems—and introduces both in-distribution and out-of-distribution evaluation protocols. By integrating causal variational autoencoders, cross-attention mechanisms, DiT architectures, and high-dimensional action modeling, the framework enables comprehensive assessment of predictive and generalization capabilities under complex physical interactions. Experiments reveal that current models perform well on low-dimensional geometric tasks but exhibit significant performance degradation in scenarios involving deformable contact and high-dimensional control; the proposed architecture notably enhances modeling efficacy under high-dimensional action conditions.

Abstract

Action-conditioned world models (ACWMs) have shown strong promise for video prediction and decision-making. However, existing benchmarks are largely restricted to egocentric navigation or narrow, task-specific robotics datasets, offering only limited coverage of the rich physical interactions required for generalized world understanding. We introduce ACWM-Phys, a new benchmark for evaluating action-conditioned prediction under diverse physical dynamics in a clean, controllable simulation environment with a carefully designed action space. ACWM-Phys contains training and evaluation data spanning rigid-body dynamics, kinematics, deformable-object interactions, and particle dynamics. To evaluate both interpolation and generalization, we design in-distribution and out-of-distribution protocols with controlled shifts in interaction patterns or scene configurations. By building the benchmark in a fully controllable simulator, ACWM-Phys enables precise data collection, reproducible evaluation, and systematic analysis of model capabilities for physically grounded world modeling. Through systematic experiments on ACWM-DiT, we find that OoD generalization depends not only on the physical regime but also on effective task complexity: models generalize well on visually simple, low-dimensional interactions with clear geometric structure, but suffer larger drops on deformable contacts, high-dimensional control, and complex articulated motion. This suggests that the model still relies heavily on visual appearance patterns instead of fully learning the underlying physics. Ablations show that cross-attention improves high-dimensional action conditioning, causal VAEs outperform frame-wise encoders, and larger action spaces are harder to model but can improve generalization by providing richer control signals. These findings guide the design of physically grounded world models.

Problem

• action-conditioned world models
• physical interaction
• video prediction
• generalization
• benchmark

Innovation

• action-conditioned world models
• physical interaction
• video prediction
• out-of-distribution generalization
• controllable simulation