Leveling3D: Leveling Up 3D Reconstruction with Feed-Forward 3D Gaussian Splatting and Geometry-Aware Generation

2026-03-17

AI Summary

This work addresses the rendering artifacts and holes in out-of-distribution view extrapolation caused by geometric incompleteness in feed-forward 3D reconstruction. To this end, the authors propose a synergistic optimization framework that integrates 3D Gaussian splatting with geometry-aware generation. By incorporating a lightweight geometry alignment adapter, a palette-filtered training strategy, and test-time mask refinement, the method effectively reconstructs missing regions in extrapolated views while preserving geometric consistency, and feeds the enhanced results back into the reconstruction pipeline. The approach achieves state-of-the-art performance on novel view synthesis and depth estimation tasks, significantly improving the quality of extrapolated views and the overall completeness of 3D reconstructions.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Learning to Search
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Feed-forward 3D reconstruction has revolutionized 3D vision, providing a powerful baseline for downstream tasks such as novel-view synthesis with 3D Gaussian Splatting. Previous works explore fixing the corrupted rendering results with a diffusion model. However, they lack geometric concern and fail at filling the missing area on the extrapolated view. In this work, we introduce Leveling3D, a novel pipeline that integrates feed-forward 3D reconstruction with geometrical-consistent generation to enable holistic simultaneous reconstruction and generation. We propose a geometry-aware leveling adapter, a lightweight technique that aligns internal knowledge in the diffusion model with the geometry prior from the feed-forward model. The leveling adapter enables generation on the artifact area of the extrapolated novel views caused by underconstrained regions of the 3D representation. Specifically, to learn a more diverse distributed generation, we introduce the palette filtering strategy for training, and a test-time masking refinement to prevent messy boundaries along the fixing regions. More importantly, the enhanced extrapolated novel views from Leveling3D could be used as the inputs for feed-forward 3DGS, leveling up the 3D reconstruction. We achieve SOTA performance on public datasets, including tasks such as novel-view synthesis and depth estimation.

Problem

• 3D reconstruction
• novel-view synthesis
• geometry-aware generation
• 3D Gaussian Splatting
• extrapolated views

Innovation

• Feed-forward 3D reconstruction
• 3D Gaussian Splatting
• Geometry-aware generation
• Leveling adapter
• Novel-view synthesis