3R-GS: Best Practice in Optimizing Camera Poses Along with 3DGS

2025-04-05

AI Summary

3D Gaussian Splatting (3DGS) suffers from strong dependence on Structure-from-Motion (SfM)-estimated camera poses and poor robustness in texture-deprived scenes. Method: We propose the first end-to-end framework jointly optimizing 3D Gaussian parameters and camera poses. Leveraging MASt3R-SfM’s large-scale geometric reconstruction priors as explicit geometric guidance, we design a robust, initialization-insensitive, multi-stage optimization strategy to mitigate the local minima trapping and weak global convergence inherent in conventional joint optimization. The method enables co-refinement of SfM poses and Gaussian attributes via differentiable rendering. Results: On standard benchmarks, our approach achieves PSNR >33.2, reduces camera rotation and translation errors by 41% and 37%, respectively, and operates at real-time inference speed—significantly outperforming the canonical two-stage SfM+3DGS pipeline.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: State Estimation
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

3D Gaussian Splatting (3DGS) has revolutionized neural rendering with its efficiency and quality, but like many novel view synthesis methods, it heavily depends on accurate camera poses from Structure-from-Motion (SfM) systems. Although recent SfM pipelines have made impressive progress, questions remain about how to further improve both their robust performance in challenging conditions (e.g., textureless scenes) and the precision of camera parameter estimation simultaneously. We present 3R-GS, a 3D Gaussian Splatting framework that bridges this gap by jointly optimizing 3D Gaussians and camera parameters from large reconstruction priors MASt3R-SfM. We note that naively performing joint 3D Gaussian and camera optimization faces two challenges: the sensitivity to the quality of SfM initialization, and its limited capacity for global optimization, leading to suboptimal reconstruction results. Our 3R-GS, overcomes these issues by incorporating optimized practices, enabling robust scene reconstruction even with imperfect camera registration. Extensive experiments demonstrate that 3R-GS delivers high-quality novel view synthesis and precise camera pose estimation while remaining computationally efficient. Project page: https://zsh523.github.io/3R-GS/

Problem

• Optimizing camera poses with 3D Gaussian Splatting for neural rendering
• Improving robustness in textureless scenes and camera parameter precision
• Jointly optimizing 3D Gaussians and camera parameters from SfM priors

Innovation

• Jointly optimizes 3D Gaussians and camera parameters
• Uses large reconstruction priors MASt3R-SfM
• Enables robust reconstruction with imperfect camera registration