Flow Distillation Sampling: Regularizing 3D Gaussians with Pre-trained Matching Priors

2025-02-11

AI Summary

3D Gaussian Splatting (3DGS) achieves remarkable rendering quality and speed but suffers from geometric distortions, holes, and floating artifacts under sparse-view settings due to the lack of explicit geometric constraints. To address this, we propose FlowDistill: the first method to leverage pre-trained matching priors (e.g., LoFTR) for unsupervised geometric regularization by analytically aligning radiance flow across views. We introduce Flow Distillation Sampling—a novel sampling paradigm that enforces optical flow consistency to guide 3DGS optimization. Our approach jointly optimizes depth, radiance fields, and surface meshes without requiring ground-truth depth or camera pose supervision. Extensive experiments demonstrate state-of-the-art performance across depth estimation, mesh reconstruction, and novel-view synthesis. On benchmarks including DTU and ScanNet, FlowDistill significantly improves geometric accuracy and cross-view rendering consistency over prior arts.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Learning with Manifolds

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

3D Gaussian Splatting (3DGS) has achieved excellent rendering quality with fast training and rendering speed. However, its optimization process lacks explicit geometric constraints, leading to suboptimal geometric reconstruction in regions with sparse or no observational input views. In this work, we try to mitigate the issue by incorporating a pre-trained matching prior to the 3DGS optimization process. We introduce Flow Distillation Sampling (FDS), a technique that leverages pre-trained geometric knowledge to bolster the accuracy of the Gaussian radiance field. Our method employs a strategic sampling technique to target unobserved views adjacent to the input views, utilizing the optical flow calculated from the matching model (Prior Flow) to guide the flow analytically calculated from the 3DGS geometry (Radiance Flow). Comprehensive experiments in depth rendering, mesh reconstruction, and novel view synthesis showcase the significant advantages of FDS over state-of-the-art methods. Additionally, our interpretive experiments and analysis aim to shed light on the effects of FDS on geometric accuracy and rendering quality, potentially providing readers with insights into its performance. Project page: https://nju-3dv.github.io/projects/fds

Problem

• Enhance 3D Gaussian Splatting geometric reconstruction
• Integrate pre-trained matching priors for accuracy
• Improve rendering in unobserved view regions

Innovation

• Leverages pre-trained geometric knowledge
• Employs strategic sampling technique
• Guides flow with optical flow