Evaluating Fisheye-Compatible 3D Gaussian Splatting Methods on Real Images Beyond 180 Degree Field of View

2025-08-09

AI Summary

This work addresses the failure of 3D Gaussian Splatting (3DGS) reconstruction on ultra-wide-field-of-view (>180°) fisheye images due to extreme radial distortion. We present the first systematic empirical evaluation of two fisheye-compatible methods—Fisheye-GS and 3DGUT—in real-world ultra-wide-angle scenarios. To overcome the unreliability of traditional Structure-from-Motion (SfM) under severe distortion, we propose a novel initialization strategy based on UniK3D—a learned monocular depth predictor—that jointly leverages multi-view geometry and single-image depth priors for robust reconstruction from sparse inputs. Experiments demonstrate that 3DGUT achieves stable visual quality across the full 200° field of view, whereas Fisheye-GS yields superior reconstruction within 160°. Our learned initialization matches SfM’s pose accuracy even under challenging conditions (e.g., fog, glare), significantly enhancing the practicality and generalizability of wide-field 3D reconstruction.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: State Estimation
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

We present the first evaluation of fisheye-based 3D Gaussian Splatting methods, Fisheye-GS and 3DGUT, on real images with fields of view exceeding 180 degree. Our study covers both indoor and outdoor scenes captured with 200 degree fisheye cameras and analyzes how each method handles extreme distortion in real world settings. We evaluate performance under varying fields of view (200 degree, 160 degree, and 120 degree) to study the tradeoff between peripheral distortion and spatial coverage. Fisheye-GS benefits from field of view (FoV) reduction, particularly at 160 degree, while 3DGUT remains stable across all settings and maintains high perceptual quality at the full 200 degree view. To address the limitations of SfM-based initialization, which often fails under strong distortion, we also propose a depth-based strategy using UniK3D predictions from only 2-3 fisheye images per scene. Although UniK3D is not trained on real fisheye data, it produces dense point clouds that enable reconstruction quality on par with SfM, even in difficult scenes with fog, glare, or sky. Our results highlight the practical viability of fisheye-based 3DGS methods for wide-angle 3D reconstruction from sparse and distortion-heavy image inputs.

Problem

• Evaluating 3D Gaussian Splatting for fisheye images beyond 180° FoV
• Analyzing distortion handling in real-world indoor and outdoor scenes
• Proposing depth-based initialization to overcome SfM limitations

Innovation

• Evaluates fisheye 3D Gaussian Splatting beyond 180° FoV
• Proposes depth-based initialization using UniK3D predictions
• Analyzes distortion tradeoffs in varying fisheye fields