NeRF-VIO: Map-Based Visual-Inertial Odometry with Initialization Leveraging Neural Radiance Fields

2025-03-11

AI Summary

To address the drift susceptibility and poor initialization robustness of visual-inertial odometry (VIO) in context-aware applications such as augmented reality (AR), this paper proposes NeRF-VIO: a visual-inertial odometry framework incorporating neural radiance field (NeRF) prior maps. Methodologically, we introduce, for the first time, a geometrically consistent NeRF initialization loss formulated directly on the SE(3) manifold, and design a two-stage update mechanism integrated into the multi-state constraint Kalman filter (MSCKF) framework to enable real-time, high-accuracy state estimation under map constraints. Our approach synergistically combines NeRF rendering, MLP-based scene representation, and Lie-group geometric modeling. Experiments on real-world AR datasets demonstrate that NeRF-VIO significantly outperforms existing NeRF-initialized methods and standard MSCKF in both localization accuracy and computational efficiency, achieving state-of-the-art performance across all evaluated sequences.

Technology Category

• Intelligent Robots: State Estimation
• Computer Vision: Motion & Tracking
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Location- and context-aware Web and WoT applications and services
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

A prior map serves as a foundational reference for localization in context-aware applications such as augmented reality (AR). Providing valuable contextual information about the environment, the prior map is a vital tool for mitigating drift. In this paper, we propose a map-based visual-inertial localization algorithm (NeRF-VIO) with initialization using neural radiance fields (NeRF). Our algorithm utilizes a multilayer perceptron model and redefines the loss function as the geodesic distance on (SE(3)), ensuring the invariance of the initialization model under a frame change within (mathfrak{se}(3)). The evaluation demonstrates that our model outperforms existing NeRF-based initialization solution in both accuracy and efficiency. By integrating a two-stage update mechanism within a multi-state constraint Kalman filter (MSCKF) framework, the state of NeRF-VIO is constrained by both captured images from an onboard camera and rendered images from a pre-trained NeRF model. The proposed algorithm is validated using a real-world AR dataset, the results indicate that our two-stage update pipeline outperforms MSCKF across all data sequences.

Problem

• Develops NeRF-VIO for visual-inertial odometry with NeRF initialization.
• Enhances localization accuracy using a two-stage update mechanism.
• Improves AR applications by integrating pre-trained NeRF models.

Innovation

• NeRF-VIO integrates neural radiance fields for initialization
• Uses geodesic distance on SE(3) for loss function
• Two-stage update mechanism in MSCKF framework