PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting

2024-10-29

arXiv.org

AI Summary

To address novel view synthesis from real-world images with sparse or unavailable camera poses, this paper proposes a single-pass feedforward method that eliminates 3D Gaussian Splatting’s reliance on accurate pose estimates and highly overlapping multi-view inputs. Methodologically, we introduce a geometry-aware confidence-guided conditional prediction mechanism for Gaussian parameters and design a lightweight, learnable module to jointly optimize coarse depth and relative pose estimation. Our framework integrates pre-trained monocular depth estimation, visual correspondence learning, differentiable Gaussian rendering, and geometric confidence modeling. Evaluated on large-scale real-world datasets, our approach achieves state-of-the-art performance, significantly improving both 3D reconstruction fidelity and novel view synthesis stability in pose-free or pose-sparse scenarios.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

We consider the problem of novel view synthesis from unposed images in a single feed-forward. Our framework capitalizes on fast speed, scalability, and high-quality 3D reconstruction and view synthesis capabilities of 3DGS, where we further extend it to offer a practical solution that relaxes common assumptions such as dense image views, accurate camera poses, and substantial image overlaps. We achieve this through identifying and addressing unique challenges arising from the use of pixel-aligned 3DGS: misaligned 3D Gaussians across different views induce noisy or sparse gradients that destabilize training and hinder convergence, especially when above assumptions are not met. To mitigate this, we employ pre-trained monocular depth estimation and visual correspondence models to achieve coarse alignments of 3D Gaussians. We then introduce lightweight, learnable modules to refine depth and pose estimates from the coarse alignments, improving the quality of 3D reconstruction and novel view synthesis. Furthermore, the refined estimates are leveraged to estimate geometry confidence scores, which assess the reliability of 3D Gaussian centers and condition the prediction of Gaussian parameters accordingly. Extensive evaluations on large-scale real-world datasets demonstrate that PF3plat sets a new state-of-the-art across all benchmarks, supported by comprehensive ablation studies validating our design choices.

Problem

• Novel view synthesis from unposed images in single feed-forward
• Addressing misaligned 3D Gaussians causing noisy/sparse gradients
• Improving 3D reconstruction without dense views or accurate poses

Innovation

• Uses monocular depth for coarse alignment
• Lightweight modules refine depth and pose
• Geometry confidence scores condition predictions