GS-CPR: Efficient Camera Pose Refinement via 3D Gaussian Splatting

2024-08-20

AI Summary

This work addresses single-image camera pose refinement under challenging outdoor illumination conditions using RGB inputs. We propose GS-CPR, a novel framework that leverages 3D Gaussian Splatting (3DGS) as the scene representation—eliminating the need for trained feature extractors or descriptors—and refines coarse poses via a single forward pass. Key contributions include: (i) the first end-to-end 2D matching pipeline integrating 3DGS with MASt3R; (ii) an exposure-adaptive module to enhance robustness to outdoor lighting variations; and (iii) a training-free, unified pose optimization workflow. Evaluated across multiple indoor and outdoor benchmarks, GS-CPR significantly outperforms NeRF-based optimization methods in both pose accuracy and inference speed, achieving state-of-the-art (SOTA) performance. Notably, it establishes new best results on two indoor datasets.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We leverage 3D Gaussian Splatting (3DGS) as a scene representation and propose a novel test-time camera pose refinement (CPR) framework, GS-CPR. This framework enhances the localization accuracy of state-of-the-art absolute pose regression and scene coordinate regression methods. The 3DGS model renders high-quality synthetic images and depth maps to facilitate the establishment of 2D-3D correspondences. GS-CPR obviates the need for training feature extractors or descriptors by operating directly on RGB images, utilizing the 3D foundation model, MASt3R, for precise 2D matching. To improve the robustness of our model in challenging outdoor environments, we incorporate an exposure-adaptive module within the 3DGS framework. Consequently, GS-CPR enables efficient one-shot pose refinement given a single RGB query and a coarse initial pose estimation. Our proposed approach surpasses leading NeRF-based optimization methods in both accuracy and runtime across indoor and outdoor visual localization benchmarks, achieving new state-of-the-art accuracy on two indoor datasets. The project page is available at https://gsloc.active.vision.

Problem

• Enhance camera pose accuracy
• Improve robustness outdoor environments
• Surpass NeRF-based optimization methods

Innovation

• 3D Gaussian Splatting scene representation
• Test-time camera pose refinement
• Exposure-adaptive module integration