Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistency

2025-03-26

AI Summary

Existing single-image-to-4D scene generation methods suffer from limited object-level modeling capacity or reliance on large-scale multi-view video training, resulting in poor generalization and high data acquisition costs. This paper introduces the first zero-shot framework for spatiotemporally consistent 4D scene generation from a single input image. Methodologically: (i) we distill multimodal foundation models to jointly enable image-to-video diffusion and 4D geometric initialization; (ii) we propose a novel prompt-guided denoising strategy coupled with latent-space temporal replacement, ensuring cross-view and cross-temporal consistency; (iii) we design a modulation-based feature refinement module to suppress generation inconsistencies. Experiments demonstrate real-time, controllable rendering of novel views and novel time steps, significantly improving generalization and inference efficiency while alleviating the critical bottleneck of 4D data scarcity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present Free4D, a novel tuning-free framework for 4D scene generation from a single image. Existing methods either focus on object-level generation, making scene-level generation infeasible, or rely on large-scale multi-view video datasets for expensive training, with limited generalization ability due to the scarcity of 4D scene data. In contrast, our key insight is to distill pre-trained foundation models for consistent 4D scene representation, which offers promising advantages such as efficiency and generalizability. 1) To achieve this, we first animate the input image using image-to-video diffusion models followed by 4D geometric structure initialization. 2) To turn this coarse structure into spatial-temporal consistent multiview videos, we design an adaptive guidance mechanism with a point-guided denoising strategy for spatial consistency and a novel latent replacement strategy for temporal coherence. 3) To lift these generated observations into consistent 4D representation, we propose a modulation-based refinement to mitigate inconsistencies while fully leveraging the generated information. The resulting 4D representation enables real-time, controllable rendering, marking a significant advancement in single-image-based 4D scene generation.

Problem

• Generates 4D scenes from single images without tuning
• Overcomes limitations of object-level and data-scarce methods
• Ensures spatial-temporal consistency in multiview video generation

Innovation

• Distill pre-trained models for 4D representation
• Adaptive guidance for spatial-temporal consistency
• Modulation-based refinement for coherent 4D rendering