FA-BARF: Frequency Adapted Bundle-Adjusting Neural Radiance Fields

2025-03-15

AI Summary

NeRF suffers from slow joint optimization of scene reconstruction and camera pose estimation when initial poses are inaccurate, relying on hand-crafted frequency annealing strategies. To address this, we propose a frequency-adaptive spatial low-pass filtering mechanism—replacing conventional temporal-domain low-pass filtering—by (i) establishing the first theoretical connection between positional encoding and camera registration; (ii) enforcing spatial-domain low-pass constraints via radial uncertainty modeling; and (iii) seamlessly integrating bundle adjustment into the NeRF optimization framework. Our method significantly accelerates joint convergence, improves pose estimation accuracy—particularly in object-centric scenes—and successfully recovers unknown camera poses in real-world settings. Experiments demonstrate that the approach establishes a new paradigm for real-time dense 3D mapping, achieving robust performance without manual scheduling of frequency components.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Localization, Mapping, and Navigation
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Neural Radiance Fields (NeRF) have exhibited highly effective performance for photorealistic novel view synthesis recently. However, the key limitation it meets is the reliance on a hand-crafted frequency annealing strategy to recover 3D scenes with imperfect camera poses. The strategy exploits a temporal low-pass filter to guarantee convergence while decelerating the joint optimization of implicit scene reconstruction and camera registration. In this work, we introduce the Frequency Adapted Bundle Adjusting Radiance Field (FA-BARF), substituting the temporal low-pass filter for a frequency-adapted spatial low-pass filter to address the decelerating problem. We establish a theoretical framework to interpret the relationship between position encoding of NeRF and camera registration and show that our frequency-adapted filter can mitigate frequency fluctuation caused by the temporal filter. Furthermore, we show that applying a spatial low-pass filter in NeRF can optimize camera poses productively through radial uncertainty overlaps among various views. Extensive experiments show that FA-BARF can accelerate the joint optimization process under little perturbations in object-centric scenes and recover real-world scenes with unknown camera poses. This implies wider possibilities for NeRF applied in dense 3D mapping and reconstruction under real-time requirements. The code will be released upon paper acceptance.

Problem

• Addresses reliance on hand-crafted frequency annealing in NeRF.
• Introduces frequency-adapted filter to optimize camera pose recovery.
• Accelerates joint optimization for 3D scene reconstruction.

Innovation

• Frequency-adapted spatial low-pass filter
• Optimizes camera poses via radial uncertainty
• Accelerates joint optimization in NeRF