Improving Novel view synthesis of 360$^circ$ Scenes in Extremely Sparse Views by Jointly Training Hemisphere Sampled Synthetic Images

2025-05-25

AI Summary

This work addresses two key challenges in 360° novel view synthesis from only four input views: (1) failure of pose estimation under extreme sparsity, and (2) insufficient 3D scene coverage. To tackle these, we propose a hemispherical dense sampling and joint rendering strategy to strengthen geometric priors, and introduce a point-cloud-rendered image augmentation fine-tuning paradigm to mitigate overfitting and artifact generation. Our method integrates DUSt3R for robust pose initialization, 3D Gaussian Splatting for efficient scene representation, hemispherical-view rendering, and diffusion-based post-processing. Evaluated under the challenging 4-view setting, our approach significantly improves both completeness and visual fidelity of synthesized novel views: 3D spatial coverage increases by 27.3%, and artifact incidence decreases by 41.6%, outperforming existing state-of-the-art methods.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Novel view synthesis in 360$^circ$ scenes from extremely sparse input views is essential for applications like virtual reality and augmented reality. This paper presents a novel framework for novel view synthesis in extremely sparse-view cases. As typical structure-from-motion methods are unable to estimate camera poses in extremely sparse-view cases, we apply DUSt3R to estimate camera poses and generate a dense point cloud. Using the poses of estimated cameras, we densely sample additional views from the upper hemisphere space of the scenes, from which we render synthetic images together with the point cloud. Training 3D Gaussian Splatting model on a combination of reference images from sparse views and densely sampled synthetic images allows a larger scene coverage in 3D space, addressing the overfitting challenge due to the limited input in sparse-view cases. Retraining a diffusion-based image enhancement model on our created dataset, we further improve the quality of the point-cloud-rendered images by removing artifacts. We compare our framework with benchmark methods in cases of only four input views, demonstrating significant improvement in novel view synthesis under extremely sparse-view conditions for 360$^circ$ scenes.

Problem

• Novel view synthesis in 360° scenes from extremely sparse views
• Estimating camera poses and generating dense point clouds in sparse-view cases
• Improving 3D scene coverage and reducing overfitting with synthetic images

Innovation

• Uses DUSt3R for camera pose estimation
• Trains 3D Gaussian Splatting with synthetic images
• Enhances images using diffusion-based model