FlexSplat: Flexible Feed-Forward 3D Gaussian Splatting without Point Cloud Correspondence

2026-08-08

AI Summary

This work addresses the challenging problem of novel view synthesis from multi-view images without calibrated cameras or known poses. The authors propose a feed-forward 3D Gaussian splatting framework that jointly trains a geometry transformer and a Gaussian decoder to end-to-end predict camera parameters and depth directly from input images, thereby constructing a compact, view-consistent Gaussian representation. Notably, this is the first query-based Gaussian reconstruction method to achieve calibration-free operation without requiring ground-truth poses or depth, while decoupling the number of Gaussians from input resolution and view count. Key innovations include depth-guided Gaussian parameterization, multi-view deformable cross-attention, and an uncertainty-weighted depth consistency loss. On ShapeNet-SRN and GSO benchmarks, the method achieves performance close to or on par with state-of-the-art pose-supervised approaches (within <0.7 dB PSNR gap) and attains the best LPIPS perceptual quality on GSO.

Abstract

We present FlexSplat, a feed-forward framework for novel view synthesis (NVS) from uncalibrated, object-centric multi-view image collections. A recent line of query-based methods reconstructs a compact set of 3D Gaussians by treating them as transformer queries that are refined with multi-view deformable attention; these methods, however, assume that camera poses are given. FlexSplat removes this assumption: a geometry transformer is trained jointly with the Gaussian decoder to predict per-image camera parameters and depth, which in turn ground a depth-guided Gaussian parameterization and a multi-view deformable cross-attention that aggregates evidence across all input views into a single, view-consistent set of primitives. An uncertainty-weighted depth-consistency objective lets the jointly trained geometry adapt to the reconstruction task, while the cross-view consensus formed during decoding absorbs the residual error of the estimated cameras and depth. The representation uses a compact Gaussian budget that is decoupled from the input resolution - unlike pixel-aligned methods, the primitive count does not grow with the image grid - and is not dictated by the number of views. On ShapeNet-SRN and Google Scanned Objects (GSO), FlexSplat matches or approaches posed state-of-the-art reconstructors while requiring neither camera poses nor ground-truth depth, and matches the best perceptual (LPIPS) quality among the compared methods on GSO. Our results indicate that a jointly trained geometry front-end is sufficient to bring calibration-free operation to query-based Gaussian reconstruction while staying within 0.7 dB PSNR of posed methods and matching their perceptual quality.

Problem

• novel view synthesis
• 3D Gaussian Splatting
• uncalibrated multi-view images
• camera pose estimation
• depth estimation

Innovation

• Gaussian Splatting
• novel view synthesis
• uncalibrated multi-view
• geometry transformer
• depth-guided representation