NeuROK: Generative 4D Neural Object Kinematics

2026-05-28

AI Summary

Existing approaches to generating realistic 4D dynamics—i.e., temporal deformations of objects under varying physical conditions—are constrained by predefined physics-based models, limiting their generalization and scalability. This work proposes Neural Object Kinematics (NeuROK), which reframes object-centric 4D dynamics as a data-driven dynamical system in a low-dimensional latent space, thereby eliminating reliance on task-specific physical priors. Built upon a Transformer encoder–decoder architecture, NeuROK learns a latent representation of object states and their mapping to deformed shapes, enabling efficient and generalizable dynamic generation across a large-scale, cross-category 4D dataset. Experiments demonstrate that NeuROK substantially outperforms current methods and significantly streamlines the pipeline for high-fidelity 4D dynamic synthesis.

Abstract

Data-driven approaches have revolutionized 3D vision, enabling transformers to effectively reconstruct and generate static 3D objects. However, generating simulative 4D dynamics -- realistic temporal deformations of static objects under various physical conditions -- remains challenging and often ad hoc, despite its importance in building comprehensive 3D world models. Most existing methods assume a predefined physical model and use system identification to estimate parameters, restricting these methods to specific categories and small-scale datasets. We propose that these restrictions can be overcome by learning a data-driven kinematic state parameterization for object-centric physical systems. Specifically, we learn both a latent space representing all possible states of the object and a decoder that maps any sampled latent to a plausibly deformed shape of the object. We refer to this parameterization as Neural Object Kinematics (NeuROK), and learn a transformer-based encoder-decoder model on a curated large-scale 4D dataset. This formulation and the learned model significantly simplify the generation of simulative dynamics since we only need to consider the dynamics within a low-dimensional latent space from the Lagrangian mechanics' perspective in classical physics. We demonstrate the effectiveness and generality of this neural simulation framework across diverse dynamic object types, showing clear advantages over prior works. Project page: https://chen-geng.com/neurok

Problem

• 4D dynamics
• object deformation
• physical simulation
• data-driven generation
• neural kinematics

Innovation

• Neural Object Kinematics
• 4D dynamics generation
• data-driven simulation
• latent space modeling
• transformer-based 4D reconstruction