SC-OmniGS: Self-Calibrating Omnidirectional Gaussian Splatting

2025-02-07

AI Summary

Traditional light field reconstruction for 360° images relies on cubemap unwrapping, perspective rectification, and pre-calibration—compromising both efficiency and accuracy. This paper introduces the first end-to-end self-calibrating spherical Gaussian splatting framework. We propose a differentiable omnidirectional camera model enabling robust pose optimization without initial estimates, and design a spherical photometric loss weighting scheme that jointly optimizes intrinsic parameters, extrinsic poses, and 3D Gaussian attributes directly on the sphere—eliminating the need for unwrapping or external calibration. Evaluated on challenging real-world panoramic sequences—including wide-baseline and non-central configurations—the method achieves superior distortion correction accuracy and novel-view synthesis quality. It delivers high-fidelity, plug-and-play omnidirectional radiance field reconstruction without manual intervention or prior geometric assumptions.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

360-degree cameras streamline data collection for radiance field 3D reconstruction by capturing comprehensive scene data. However, traditional radiance field methods do not address the specific challenges inherent to 360-degree images. We present SC-OmniGS, a novel self-calibrating omnidirectional Gaussian splatting system for fast and accurate omnidirectional radiance field reconstruction using 360-degree images. Rather than converting 360-degree images to cube maps and performing perspective image calibration, we treat 360-degree images as a whole sphere and derive a mathematical framework that enables direct omnidirectional camera pose calibration accompanied by 3D Gaussians optimization. Furthermore, we introduce a differentiable omnidirectional camera model in order to rectify the distortion of real-world data for performance enhancement. Overall, the omnidirectional camera intrinsic model, extrinsic poses, and 3D Gaussians are jointly optimized by minimizing weighted spherical photometric loss. Extensive experiments have demonstrated that our proposed SC-OmniGS is able to recover a high-quality radiance field from noisy camera poses or even no pose prior in challenging scenarios characterized by wide baselines and non-object-centric configurations. The noticeable performance gain in the real-world dataset captured by consumer-grade omnidirectional cameras verifies the effectiveness of our general omnidirectional camera model in reducing the distortion of 360-degree images.

Problem

• Self-calibrating omnidirectional Gaussian splatting
• Direct omnidirectional camera pose calibration
• Differentiable omnidirectional camera model distortion rectification

Innovation

• Direct omnidirectional camera calibration
• Differentiable omnidirectional camera model
• Joint optimization of camera and Gaussians