GSplatLoc: Grounding Keypoint Descriptors into 3D Gaussian Splatting for Improved Visual Localization

2024-09-24

arXiv.org

AI Summary

Existing visual localization methods often suffer from high optimization complexity or insufficient localization accuracy. This paper proposes a two-stage, end-to-end differentiable localization framework based on 3D Gaussian Splatting (3DGS). First, dense keypoints are extracted using XFeat, and their descriptors are explicitly embedded into the joint geometry-appearance representation of 3DGS to enable pose-free coarse pose estimation—eliminating the need for an explicit pose network. Second, photometric warping loss is employed to refine the estimated pose via differentiable rendering optimization. To our knowledge, this is the first work to directly integrate lightweight keypoint descriptors into the 3DGS representation without requiring dedicated camera pose network training. Extensive experiments on standard indoor and outdoor benchmarks demonstrate that our method significantly outperforms state-of-the-art neural rendering–based localization approaches—including NeRFMatch and PNeRFLoc—in both localization accuracy and robustness.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Localization, Mapping, and Navigation
• Search and Optimization: Local Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Location- and context-aware Web and WoT applications and services
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Although various visual localization approaches exist, such as scene coordinate regression and camera pose regression, these methods often struggle with optimization complexity or limited accuracy. To address these challenges, we explore the use of novel view synthesis techniques, particularly 3D Gaussian Splatting (3DGS), which enables the compact encoding of both 3D geometry and scene appearance. We propose a two-stage procedure that integrates dense and robust keypoint descriptors from the lightweight XFeat feature extractor into 3DGS, enhancing performance in both indoor and outdoor environments. The coarse pose estimates are directly obtained via 2D-3D correspondences between the 3DGS representation and query image descriptors. In the second stage, the initial pose estimate is refined by minimizing the rendering-based photometric warp loss. Benchmarking on widely used indoor and outdoor datasets demonstrates improvements over recent neural rendering-based localization methods, such as NeRFMatch and PNeRFLoc.

Problem

• Improves visual localization accuracy using 3D Gaussian Splatting.
• Integrates keypoint descriptors for enhanced indoor and outdoor performance.
• Refines pose estimates with photometric warp loss minimization.

Innovation

• Integrates XFeat descriptors into 3D Gaussian Splatting
• Uses 2D-3D correspondences for coarse pose estimation
• Refines pose with photometric warp loss minimization