PhysiFormer: Learning to Simulate Mechanics in World Space

2026-06-25

AI Summary

This work addresses the challenge of generating physically plausible and geometry-aware 3D motion simulations for multiple objects in world coordinates, accommodating both rigid and elastic materials as well as complex interactions. To this end, the authors propose PhysiFormer, the first diffusion-based Transformer model that directly predicts future trajectories of 3D mesh vertices in world space, eschewing traditional physics constraints and view-dependent representations. PhysiFormer employs a temporally-, spatially-, and object-wise factorized attention mechanism to achieve permutation-invariant multi-object reasoning. Taking vertex positions, velocities, and material types as input, the model leverages a denoising diffusion process to produce diverse yet physically consistent motion sequences. Experiments demonstrate that PhysiFormer significantly outperforms autoregressive baselines in trajectory accuracy, rigidity preservation, and momentum consistency, and generalizes effectively to scenarios involving mixed materials, real-world geometries, and larger numbers of interacting objects.

Abstract

We present PhysiFormer, a diffusion transformer for physically-plausible 3D object motion. Unlike video world models that operate in view-dependent pixel space, PhysiFormer represents objects as 3D meshes expressed in world coordinates. Given the initial vertex positions and velocities, as well as object material type, rigid or elastic, the model samples future vertex trajectories. While related neural physics approaches build on ad-hoc latent spaces or explicitly enforce rigidity and causality, PhysiFormer shows that excellent results can be obtained without any such inductive biases, by casting vertex trajectory prediction as a single denoising diffusion process directly in world coordinates. The probabilistic formulation captures uncertainty in the learned dynamics, enabling diverse plausible futures from initial conditions, making this framework potentially useful for applications with unobserved uncertainty. The model features attention factorised over time, space, and objects for efficiency, enabling permutation-invariant multi-object reasoning without needing explicit object encoding. Trained on over 100k simulated trajectories, PhysiFormer generates rigid and elastic mechanics, and generalises to mixed-material settings, unseen real-world geometries, and larger object counts. It substantially outperforms autoregressive baselines in trajectory accuracy, rigidity preservation, and momentum-based physical consistency. Our results position coordinate-space diffusion as a promising step toward view-invariant, geometry-aware world modelling for robotics, graphics, and physical design. Visualisations, code, and models are available at https://yimingc9.github.io/physiformer.

Problem

• 3D object motion
• physical plausibility
• world coordinates
• vertex trajectory prediction
• uncertainty modeling

Innovation

• diffusion transformer
• world-space simulation
• 3D mesh dynamics
• inductive bias-free learning
• multi-object reasoning