FreeGen: Feed-Forward Reconstruction-Generation Co-Training for Free-Viewpoint Driving Scene Synthesis

2025-12-04

AI Summary

Existing autonomous driving simulation datasets and generation methods struggle to simultaneously ensure interpolation consistency under trajectory-deviated viewpoints and extrapolation realism, hindering closed-loop simulation and large-scale pretraining. To address this, we propose a reconstruction–generation co-training framework that jointly optimizes geometric stability and visual realism via bidirectional knowledge distillation. We introduce a differentiable neural rendering module coupled with a geometry-aware generative network, integrating structured geometric representations with generative priors. Furthermore, we adopt a feed-forward co-training mechanism enabling generalizable free-view synthesis without scene-specific fine-tuning. Our approach achieves state-of-the-art performance across multiple benchmarks, significantly improving cross-view consistency and extrapolation realism. It provides high-quality, high-fidelity, and strongly generalizable synthetic data—advancing scalable autonomous driving simulation and foundation model pretraining.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Generation

Application Category

• Web Mining and Content Analysis: Web data generation and simulation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Closed-loop simulation and scalable pre-training for autonomous driving require synthesizing free-viewpoint driving scenes. However, existing datasets and generative pipelines rarely provide consistent off-trajectory observations, limiting large-scale evaluation and training. While recent generative models demonstrate strong visual realism, they struggle to jointly achieve interpolation consistency and extrapolation realism without per-scene optimization. To address this, we propose FreeGen, a feed-forward reconstruction-generation co-training framework for free-viewpoint driving scene synthesis. The reconstruction model provides stable geometric representations to ensure interpolation consistency, while the generation model performs geometry-aware enhancement to improve realism at unseen viewpoints. Through co-training, generative priors are distilled into the reconstruction model to improve off-trajectory rendering, and the refined geometry in turn offers stronger structural guidance for generation. Experiments demonstrate that FreeGen achieves state-of-the-art performance for free-viewpoint driving scene synthesis.

Problem

• Synthesizes free-viewpoint driving scenes for simulation and pre-training
• Ensures interpolation consistency and extrapolation realism without per-scene optimization
• Improves off-trajectory rendering and geometry-aware enhancement through co-training

Innovation

• Reconstruction-generation co-training for viewpoint synthesis
• Geometry-aware enhancement improves unseen viewpoint realism
• Generative priors distill into reconstruction for off-trajectory rendering