3D Gaussian Flats: Hybrid 2D/3D Photometric Scene Reconstruction

2025-09-19

AI Summary

To address geometric inaccuracies and translucency artifacts in texture-sparse planar regions during radiance field reconstruction, this paper proposes a 2D/3D Gaussian hybrid representation: planar-constrained 2D Gaussians model flat surfaces, while deformable 3D Gaussians capture complex geometry; both are jointly optimized within a novel differentiable rendering framework. Our method incorporates plane-aware detection, depth regularization, and photometric reconstruction loss in an end-to-end optimization pipeline, achieving high geometric precision without sacrificing visual fidelity. Evaluated on ScanNet++ and ScanNetv2, it sets new state-of-the-art performance in depth estimation. The reconstructed meshes exhibit superior completeness and realism, enabling high-fidelity, non-overfitting novel-view synthesis and robust digital twin construction.

Technology Category

• Computer Vision: 3D Computer Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advances in radiance fields and novel view synthesis enable creation of realistic digital twins from photographs. However, current methods struggle with flat, texture-less surfaces, creating uneven and semi-transparent reconstructions, due to an ill-conditioned photometric reconstruction objective. Surface reconstruction methods solve this issue but sacrifice visual quality. We propose a novel hybrid 2D/3D representation that jointly optimizes constrained planar (2D) Gaussians for modeling flat surfaces and freeform (3D) Gaussians for the rest of the scene. Our end-to-end approach dynamically detects and refines planar regions, improving both visual fidelity and geometric accuracy. It achieves state-of-the-art depth estimation on ScanNet++ and ScanNetv2, and excels at mesh extraction without overfitting to a specific camera model, showing its effectiveness in producing high-quality reconstruction of indoor scenes.

Problem

• Reconstructing flat texture-less surfaces without uneven semi-transparent artifacts
• Balancing visual quality with geometric accuracy in photometric scene reconstruction
• Creating high-quality 3D reconstructions that work across different camera models

Innovation

• Hybrid 2D/3D Gaussian representation for scene reconstruction
• Jointly optimizes constrained planar and freeform Gaussians
• Dynamically detects and refines planar regions for accuracy