Perceptual 3D Simulation With Physical World Modeling

2026-06-25

AI Summary

Accurately predicting the post-manipulation evolution of real-world scenes from partially observed inputs and incomplete 3D transformation cues remains a fundamental challenge. This work proposes P3Sim, a system that integrates a learned physical world model, a geometry-conditioning module, and temporally consistent persistent scene memory. By combining probabilistic generative modeling with explicit geometric structure, P3Sim strikes a balance between data-driven flexibility and inductive biases. The approach demonstrates strong multi-task generalization, achieving state-of-the-art performance in novel view synthesis, object manipulation, and dynamic scene prediction, thereby significantly advancing the capabilities of general-purpose 3D scene understanding and transformation.

Abstract

Predicting how a scene will evolve after a desired 3D transformation from images is a central goal in vision, graphics, and robotics. Yet unlike ideal simulators with full access to 3D geometry and dynamics, real world systems must rely on perceptual inputs and local actions that are inherently partial and incomplete. In this work, we present P3Sim, a physical world modeling system that simulates future scene states under both partial observations and incomplete 3D transformation signals. P3Sim is composed of three interacting components: a learned physical world model, a geometric conditioning module, and a persistent scene memory. The world model interprets perception as probabilistic inference over multimodal scene variables, providing predictions of the distributions of any scene variable conditioned on any combination of others. The geometric conditioning module provides a partial 3D transform signal for conditioning the world model at inference time. The persistent scene memory integrates predictions over time, enabling online updates and consistency under uncertainty. By combining learned inference with explicit geometric structure, P3Sim balances data-driven flexibility with built-in inductive bias. This design yields a flexible perceptual simulator that generalizes across diverse 3D transformation tasks, such as novel view synthesis, object manipulation, and dynamic scene prediction, advancing toward general purpose 3D scene understanding and transformation.

Problem

• perceptual simulation
• 3D transformation
• partial observation
• physical world modeling
• scene prediction

Innovation

• perceptual simulation
• physical world modeling
• 3D transformation
• probabilistic inference
• scene memory