ST-Gen4D: Embedding 4D Spatiotemporal Cognition into World Model for 4D Generation

2026-05-08

AI Summary

Existing 4D generation methods struggle to simultaneously ensure global appearance consistency and local dynamic realism, often lacking deep modeling of physical spatiotemporal regularities. This work proposes the first generative framework embedded with a 4D spatiotemporal cognitive mechanism: it constructs global appearance and local dynamic graphs from multimodal features, employs a semantic bridging fusion strategy to form a unified 4D cognitive graph, and integrates a world model to reason future states, thereby guiding a latent diffusion model to generate structurally plausible and topologically consistent 4D Gaussian scenes. Evaluated on both a newly curated and integrated ST-4D dataset, the method significantly improves geometric plausibility and spatiotemporal coherence in both 3D and 4D generation tasks.

Abstract

Generative models have achieved success in producing apparently coherent 2D videos, but remain challenging in the physical world due to lack of 4D spatiotemporal scale. Typically, existing 4D generative models directly embed macro scale constraints to enhance overall spatiotemporal consistency. However, these methods only ensure global appearance coherence and fail to reveal the local dynamics of the physical world. Our insight is that global appearance structure and local dynamic topology empower 4D spatiotemporal cognition, thereby enabling 4D generation with spatiotemporal regularities. In this work, we propose ST-Gen4D, a 4D generation framework with 4D spatiotemporal cognition-based world model. Our model is guided by four key designs: 1) Spatiotemporal representation. We encode various modalities into multiple representations as a feature basis. 2) Spatiotemporal cognition. We sculpture these representations into global appearance graph and local dynamic graph, and fuse them via semantic-bridged spatiotemporal fusion to obtain a 4D cognition graph. 3) Spatiotemporal reasoning. We utilize a world model to derive future state based on the 4D cognition. 4) Spatiotemporal generation. We leverage the derived cognition as condition to guide latent diffusion for 4D Gaussian generation. By deeply integrating 4D intrinsic cognition with generative priors, our model guarantees the structural rationality and topological consistency of 4D generation. Moreover, we propose ST-4D datasets by aggregating public 4D datasets and self-built subset. Extensive experiments demonstrate the superiority of our ST-Gen4D across 3D and 4D generation tasks.

Problem

• 4D generation
• spatiotemporal consistency
• local dynamics
• global appearance
• world model

Innovation

• 4D generation
• spatiotemporal cognition
• world model
• graph fusion
• latent diffusion