From None to All: Self-Supervised 3D Reconstruction via Novel View Synthesis

2026-03-28

AI Summary

This work proposes NAS3R, a novel framework that achieves end-to-end self-supervised 3D Gaussian reconstruction from uncalibrated, pose-free 2D images without requiring ground-truth 3D annotations or pretrained priors. NAS3R jointly learns explicit 3D geometry and camera parameters through a shared Transformer backbone, incorporating a mask attention mechanism and depth-guided 3D Gaussian representations, optimized solely with 2D photometric supervision. Experiments demonstrate that NAS3R outperforms existing self-supervised methods across multiple benchmarks, attaining geometric accuracy comparable to supervised approaches, while remaining flexible enough to integrate known intrinsics or prior information when available.

Abstract

In this paper, we introduce NAS3R, a self-supervised feed-forward framework that jointly learns explicit 3D geometry and camera parameters with no ground-truth annotations and no pretrained priors. During training, NAS3R reconstructs 3D Gaussians from uncalibrated and unposed context views and renders target views using its self-predicted camera parameters, enabling self-supervised training from 2D photometric supervision. To ensure stable convergence, NAS3R integrates reconstruction and camera prediction within a shared transformer backbone regulated by masked attention, and adopts a depth-based Gaussian formulation that facilitates well-conditioned optimization. The framework is compatible with state-of-the-art supervised 3D reconstruction architectures and can incorporate pretrained priors or intrinsic information when available. Extensive experiments show that NAS3R achieves superior results to other self-supervised methods, establishing a scalable and geometry-aware paradigm for 3D reconstruction from unconstrained data. Code and models are publicly available at https://ranrhuang.github.io/nas3r/.

Problem

• self-supervised
• 3D reconstruction
• novel view synthesis
• camera estimation
• unconstrained images

Innovation

• self-supervised
• 3D reconstruction
• novel view synthesis
• 3D Gaussians
• camera parameter estimation