Beyond Pixels: From Video Priors to 4D Worlds

2026-08-11

AI Summary

Existing 4D generation methods struggle to achieve generalizable, high-quality dynamic 3D scene synthesis due to distribution mismatches, error propagation, or reliance on specific video generators. This work proposes a novel paradigm that directly generates 4D scenes from the final denoised latent variables of off-the-shelf video diffusion models. By aligning these latents with a token grid from a pretrained 4D decoder and optimizing them through frame-level and global spatiotemporal attention mechanisms, the approach bypasses conventional RGB reconstruction stages. It leverages—for the first time—the shared VAE latent space of video diffusion models as a universal interface, enabling cross-model 4D prediction without retraining. On the Text4D-200 and I4D-200 benchmarks, the method improves DINO-F1 scores by 2.88–3.45 and 5.81 points, respectively, and achieves significantly higher human ratings in geometric accuracy, temporal consistency, and overall quality compared to existing approaches.

Abstract

4D generation synthesizes dynamic 3D scenes from conditions such as text or images. Existing methods either reconstruct generated RGB videos with a separate 4D model or adapt a particular video generator to predict geometry directly. The former suffers from distribution mismatch and error propagation, whereas the latter ties 4D prediction to a specific generator and may require retraining when the generator or conditioning regime changes. We ask whether the final denoised latents of video models that share a variational autoencoder (VAE) can instead provide a reusable interface to explicit 4D prediction. Building on this insight, we introduce direct latent-to-4D generation and instantiate it as Latent-to-4D, which bypasses RGB by aligning a video latent with the token grid of a pretrained 4D decoder and refining it through frame-wise and global spatiotemporal attention. Trained on roughly 1K existing reconstruction clips, a single checkpoint transfers unchanged across multiple video diffusion transformers within the same VAE family. On Text4D-200 and I4D-200, Latent-to-4D surpasses matched same-latent Wan+4RC cascades in projection-based DINO-F1 by 2.88--3.45 and 5.81 points, respectively, while also being preferred by human raters for geometry, temporal stability, and overall quality.

Problem

• 4D generation
• video priors
• distribution mismatch
• error propagation
• generator decoupling

Innovation

• Latent-to-4D
• 4D generation
• video diffusion
• spatiotemporal attention
• VAE latent alignment