CAD-NeRF: Learning NeRFs from Uncalibrated Few-view Images by CAD Model Retrieval

2024-11-05

Frontiers Comput. Sci.

AI Summary

This work addresses the challenging problem of neural radiance field (NeRF) reconstruction from extremely sparse views (<10 images) without camera calibration. We propose a CAD-guided self-supervised framework: first, geometrically similar CAD models are retrieved from ShapeNet to provide coarse pose estimates and structural priors; second, a multi-view collaborative pose retrieval mechanism is introduced to mitigate cross-view pose inconsistency; finally, a deformable density field and camera poses are jointly optimized to enable CAD-guided density prior modeling and self-supervised NeRF training. Our method significantly improves geometric fidelity and generalization on both synthetic and real-world data. To the best of our knowledge, this is the first end-to-end self-supervised NeRF reconstruction approach leveraging CAD model retrieval—requiring neither pose initialization nor manual annotations.

Technology Category

• Computer Vision: Image and Video Retrieval
• Machine Learning: Unsupervised & Self-Supervised Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Reconstructing from multi-view images is a longstanding problem in 3D vision, where neural radiance fields (NeRFs) have shown great potential and get realistic rendered images of novel views. Currently, most NeRF methods either require accurate camera poses or a large number of input images, or even both. Reconstructing NeRF from few-view images without poses is challenging and highly ill-posed. To address this problem, we propose CAD-NeRF, a method reconstructed from less than 10 images without any known poses. Specifically, we build a mini library of several CAD models from ShapeNet and render them from many random views. Given sparse-view input images, we run a model and pose retrieval from the library, to get a model with similar shapes, serving as the density supervision and pose initializations. Here we propose a multi-view pose retrieval method to avoid pose conflicts among views, which is a new and unseen problem in uncalibrated NeRF methods. Then, the geometry of the object is trained by the CAD guidance. The deformation of the density field and camera poses are optimized jointly. Then texture and density are trained and fine-tuned as well. All training phases are in self-supervised manners. Comprehensive evaluations of synthetic and real images show that CAD-NeRF successfully learns accurate densities with a large deformation from retrieved CAD models, showing the generalization abilities.

Problem

• Reconstructing NeRF from few uncalibrated images
• Addressing pose conflicts in uncalibrated NeRF methods
• Enhancing geometry with CAD model guidance

Innovation

• Retrieves CAD models for shape and pose initialization
• Uses multi-view pose retrieval to avoid conflicts
• Jointly optimizes density field deformation and poses