SelfSplat: Pose-Free and 3D Prior-Free Generalizable 3D Gaussian Splatting

2024-11-26

arXiv.org

AI Summary

This work addresses 3D reconstruction from unordered multi-view images without camera pose annotations or 3D priors. We propose the first end-to-end, generalizable 3D Gaussian Splatting (3DGS) reconstruction framework for this challenging setting. Our method introduces a matching-aware, self-supervised pose estimation and depth refinement module that jointly optimizes camera poses and geometry through mutual reinforcement—eliminating reliance on pose supervision or 3D ground truth. It integrates explicit 3D Gaussian representations, self-supervised monocular depth estimation, photometric consistency constraints, differentiable rendering, and a match-guided pose network. Evaluated on RealEstate10K, ACID, and DL3DV, our approach significantly outperforms state-of-the-art methods in both geometric accuracy and appearance fidelity, while demonstrating strong zero-shot cross-dataset generalization. The code and pretrained models are publicly released.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance

Abstract

We propose SelfSplat, a novel 3D Gaussian Splatting model designed to perform pose-free and 3D prior-free generalizable 3D reconstruction from unposed multi-view images. These settings are inherently ill-posed due to the lack of ground-truth data, learned geometric information, and the need to achieve accurate 3D reconstruction without finetuning, making it difficult for conventional methods to achieve high-quality results. Our model addresses these challenges by effectively integrating explicit 3D representations with self-supervised depth and pose estimation techniques, resulting in reciprocal improvements in both pose accuracy and 3D reconstruction quality. Furthermore, we incorporate a matching-aware pose estimation network and a depth refinement module to enhance geometry consistency across views, ensuring more accurate and stable 3D reconstructions. To present the performance of our method, we evaluated it on large-scale real-world datasets, including RealEstate10K, ACID, and DL3DV. SelfSplat achieves superior results over previous state-of-the-art methods in both appearance and geometry quality, also demonstrates strong cross-dataset generalization capabilities. Extensive ablation studies and analysis also validate the effectiveness of our proposed methods. Code and pretrained models are available at https://gynjn.github.io/selfsplat/

Problem

• Pose-free 3D reconstruction from unposed images
• 3D prior-free generalizable Gaussian Splatting
• Improving geometry consistency without ground-truth data

Innovation

• Integrates 3D representations with self-supervised depth
• Uses matching-aware pose estimation network
• Incorporates depth refinement for geometry consistency