Self-Calibrating Gaussian Splatting for Large Field of View Reconstruction

2025-02-13

AI Summary

This paper addresses the challenge of jointly optimizing camera parameters, lens distortion, and 3D Gaussian representations in wide-field-of-view (FOV) image reconstruction. Methodologically: (1) it proposes a hybrid distortion modeling module integrating an invertible residual network with explicit grid-based fusion, enabling robust modeling for arbitrary ultra-wide-angle lenses; (2) it introduces a cube-map resampling strategy to preserve geometric fidelity and rendering quality under large FOV; and (3) it combines Gaussian Splatting rasterization acceleration with end-to-end optimization. Evaluated on both synthetic and real-world datasets, the method achieves state-of-the-art performance: it significantly reduces the number of required input images, outperforms conventional camera models in reconstruction accuracy, and produces high-resolution, distortion-free, dense scene representations without artifacts.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In this paper, we present a self-calibrating framework that jointly optimizes camera parameters, lens distortion and 3D Gaussian representations, enabling accurate and efficient scene reconstruction. In particular, our technique enables high-quality scene reconstruction from Large field-of-view (FOV) imagery taken with wide-angle lenses, allowing the scene to be modeled from a smaller number of images. Our approach introduces a novel method for modeling complex lens distortions using a hybrid network that combines invertible residual networks with explicit grids. This design effectively regularizes the optimization process, achieving greater accuracy than conventional camera models. Additionally, we propose a cubemap-based resampling strategy to support large FOV images without sacrificing resolution or introducing distortion artifacts. Our method is compatible with the fast rasterization of Gaussian Splatting, adaptable to a wide variety of camera lens distortion, and demonstrates state-of-the-art performance on both synthetic and real-world datasets.

Problem

• Optimizes camera parameters and lens distortion
• Enables large field-of-view scene reconstruction
• Models complex lens distortions accurately

Innovation

• Self-calibrating camera parameter optimization
• Hybrid network for lens distortion modeling
• Cubemap-based resampling for large FOV