FreeScale: Scaling 3D Scenes via Certainty-Aware Free-View Generation

2026-04-12

AI Summary

Current general-purpose novel view synthesis (NVS) models are hindered by the scarcity of large-scale, diverse real-world training data with accurate camera trajectories. This work proposes a scene reconstruction–based data augmentation approach that generates high-quality, semantically plausible, and reconstruction-error–robust novel views from limited real image sequences via a determinism-aware free-viewpoint sampling strategy. By effectively leveraging imperfect reconstructions as geometric proxies, the method avoids amplifying artifacts and enables both feed-forward NVS model training and 3D Gaussian splatting optimization. Experiments demonstrate that the proposed approach improves feed-forward NVS models by 2.7 dB in PSNR on out-of-distribution benchmarks and substantially enhances the optimization performance of 3D Gaussian splatting across multiple datasets.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

The development of generalizable Novel View Synthesis (NVS) models is critically limited by the scarcity of large-scale training data featuring diverse and precise camera trajectories. While real-world captures are photorealistic, they are typically sparse and discrete. Conversely, synthetic data scales but suffers from a domain gap and often lacks realistic semantics. We introduce FreeScale, a novel framework that leverages the power of scene reconstruction to transform limited real-world image sequences into a scalable source of high-quality training data. Our key insight is that an imperfect reconstructed scene serves as a rich geometric proxy, but naively sampling from it amplifies artifacts. To this end, we propose a certainty-aware free-view sampling strategy identifying novel viewpoints that are both semantically meaningful and minimally affected by reconstruction errors. We demonstrate FreeScale's effectiveness by scaling up the training of feedforward NVS models, achieving a notable gain of 2.7 dB in PSNR on challenging out-of-distribution benchmarks. Furthermore, we show that the generated data can actively enhance per-scene 3D Gaussian Splatting optimization, leading to consistent improvements across multiple datasets. Our work provides a practical and powerful data generation engine to overcome a fundamental bottleneck in 3D vision. Project page: https://mvp-ai-lab.github.io/FreeScale.

Problem

• Novel View Synthesis
• 3D Scene Reconstruction
• Training Data Scarcity
• Camera Trajectories
• Domain Gap

Innovation

• Novel View Synthesis
• Certainty-Aware Sampling
• 3D Scene Reconstruction
• Free-View Generation
• 3D Gaussian Splatting