PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models

2026-06-25

AI Summary

This work addresses the limitation of existing game world models in explicitly controlling physical rules—such as gravity—which hinders their applicability to creative games requiring precise physics editing. To overcome this, the authors construct a large-scale multimodal dataset based on Unreal Engine 5’s replay and rendering pipeline. By replaying identical action sequences under systematically varied gravity parameters from the same initial conditions, they generate controllable and attributable physical dynamics. The dataset is the first to explicitly annotate gravity parameters and encompasses 12 cinematic scenes, over 100 hours of interactive gameplay, and 60 million synchronized multimodal frames—including RGB, depth, normals, audio, actions, camera trajectories, semantic labels, and engine states—substantially enhancing generative models’ ability to maintain gravity consistency and edit physical rules in dynamic environments.

Abstract

Recent game world models can synthesize visually plausible, action-conditioned rollouts. However, their interaction behaviors often remain limited to exploratory or wandering trajectories, and physical dynamics are typically learned as implicit correlations from data rather than as controllable variables. This limitation hinders their applicability to authored game environments, where physical rules are deliberately designed and require explicit manipulation. We introduce PhysEditWorld, a multimodal dataset with physical parameters, with a primary focus on gravity in this initial version. At its core, PhysEditWorld is built upon a replay paradigm implemented with a UE5 replay-and-rendering pipeline. Each scenario records a normalized action trace and replays the same initial state, character controller, action sequence, and camera policy under multiple gravity configurations, enabling controlled and attributable physical variation. PhysEditWorld contains 12 cinematic UE5 scenes, over 100 hours of gameplay interactions, and more than 60 million rendered rollout frames. Each sample provides synchronized multimodal signals, including RGB, depth, normals, audio, action traces, camera trajectory, engine states, semantic annotations, and explicit gravity labels. We further conduct initial utility studies on both generative video models and world understanding models, demonstrating that PhysEditWorld enables improved gravity-faithful dynamics modeling, enhances consistency under physical edits, and provides a scalable foundation for controllable world modeling research.

Problem

• world models
• physics-editable
• controllable dynamics
• game environments
• physical parameters

Innovation

• physics-editable world models
• controllable physical dynamics
• multimodal dataset
• gravity manipulation
• UE5 replay-and-rendering pipeline