Efficient4D: Fast Dynamic 3D Object Generation from a Single-view Video

2024-01-16

AI Summary

To address the challenges of missing 4D annotations and low end-to-end optimization efficiency in monocular video-based dynamic 3D reconstruction, this work proposes a two-stage decoupled paradigm: first generating multi-view temporally consistent images via a diffusion model, then driving 4D Gaussian Splatting for explicit reconstruction. We introduce an inconsistency-aware confidence-weighted loss and a lightweight Score Distillation Sampling (SDS) loss, significantly improving robustness under sparse-view conditions. Compared to Consistent4D, our method accelerates training tenfold (10 minutes vs. 120 minutes), enables real-time continuous trajectory rendering, and achieves state-of-the-art novel-view synthesis quality. To the best of our knowledge, this is the first work to organically integrate generative modeling with explicit 4D reconstruction, establishing a new paradigm for efficient, high-fidelity dynamic scene reconstruction.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Generating dynamic 3D object from a single-view video is challenging due to the lack of 4D labeled data. An intuitive approach is to extend previous image-to-3D pipelines by transferring off-the-shelf image generation models such as score distillation sampling.However, this approach would be slow and expensive to scale due to the need for back-propagating the information-limited supervision signals through a large pretrained model. To address this, we propose an efficient video-to-4D object generation framework called Efficient4D. It generates high-quality spacetime-consistent images under different camera views, and then uses them as labeled data to directly reconstruct the 4D content through a 4D Gaussian splatting model. Importantly, our method can achieve real-time rendering under continuous camera trajectories. To enable robust reconstruction under sparse views, we introduce inconsistency-aware confidence-weighted loss design, along with a lightly weighted score distillation loss. Extensive experiments on both synthetic and real videos show that Efficient4D offers a remarkable 10-fold increase in speed when compared to prior art alternatives while preserving the quality of novel view synthesis. For example, Efficient4D takes only 10 minutes to model a dynamic object, vs 120 minutes by the previous art model Consistent4D.

Problem

• Generates dynamic 3D objects from single-view videos efficiently
• Overcomes slow and expensive 4D labeled data generation
• Achieves real-time rendering with high-quality spacetime consistency

Innovation

• Efficient video-to-4D framework with Gaussian splatting
• Real-time rendering on continuous camera trajectories
• Inconsistency-aware confidence-weighted loss design