FrugalNeRF: Fast Convergence for Few-shot Novel View Synthesis without Learned Priors

2024-10-21

arXiv.org

AI Summary

To address severe overfitting and slow convergence in few-shot Neural Radiance Fields (NeRF) for novel view synthesis, this paper proposes a multi-scale shared voxel representation framework that requires no pre-trained priors. Our method introduces three key innovations: (1) a cross-scale geometric adaptation mechanism that hierarchically aligns voxel features across resolutions; (2) a reprojection-error-based pseudo-depth selection strategy—the first unsupervised, prior-free depth guidance for NeRF; and (3) a multi-scale weight-sharing architecture coupled with an unsupervised depth-guided training paradigm. Extensive experiments on LLFF, DTU, and RealEstate-10K demonstrate substantial improvements over state-of-the-art methods such as FreeNeRF, achieving higher reconstruction fidelity and significantly reduced training time—effectively balancing efficiency and accuracy in few-shot settings.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Neural Radiance Fields (NeRF) face significant challenges in few-shot scenarios, primarily due to overfitting and long training times for high-fidelity rendering. Existing methods, such as FreeNeRF and SparseNeRF, use frequency regularization or pre-trained priors but struggle with complex scheduling and bias. We introduce FrugalNeRF, a novel few-shot NeRF framework that leverages weight-sharing voxels across multiple scales to efficiently represent scene details. Our key contribution is a cross-scale geometric adaptation scheme that selects pseudo ground truth depth based on reprojection errors across scales. This guides training without relying on externally learned priors, enabling full utilization of the training data. It can also integrate pre-trained priors, enhancing quality without slowing convergence. Experiments on LLFF, DTU, and RealEstate-10K show that FrugalNeRF outperforms other few-shot NeRF methods while significantly reducing training time, making it a practical solution for efficient and accurate 3D scene reconstruction.

Problem

• Addresses overfitting and slow training in few-shot NeRF
• Eliminates need for complex scheduling and biased priors
• Enables efficient 3D scene reconstruction with minimal data

Innovation

• Weight-sharing voxels for efficient scene representation
• Cross-scale geometric adaptation for depth selection
• Integration of pre-trained priors without slowing convergence