3D Gaussian Splatting with Normal Information for Mesh Extraction and Improved Rendering

2025-01-14

AI Summary

To address the imprecise surface reconstruction and compromised rendering realism of 3D Gaussian Splatting (3DGS) on complex geometries, this work introduces normal supervision into the differentiable 3DGS framework for the first time. We propose a signed distance function (SDF)-guided normal regularization method that jointly optimizes photometric fidelity and geometric accuracy. By unifying photometric consistency and explicit surface geometric constraints, our approach enables high-fidelity mesh extraction and real-time novel-view synthesis. Evaluated on the Mip-NeRF360 and Tanks and Temples benchmarks, our method surpasses existing mesh-based rendering approaches in photometric metrics (e.g., PSNR, SSIM), while preserving high-geometric-fidelity reconstructed meshes. This establishes a new paradigm that simultaneously delivers high-quality rendering and geometric usability—benefiting downstream applications such as video generation, AR, and VR.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Differentiable 3D Gaussian splatting has emerged as an efficient and flexible rendering technique for representing complex scenes from a collection of 2D views and enabling high-quality real-time novel-view synthesis. However, its reliance on photometric losses can lead to imprecisely reconstructed geometry and extracted meshes, especially in regions with high curvature or fine detail. We propose a novel regularization method using the gradients of a signed distance function estimated from the Gaussians, to improve the quality of rendering while also extracting a surface mesh. The regularizing normal supervision facilitates better rendering and mesh reconstruction, which is crucial for downstream applications in video generation, animation, AR-VR and gaming. We demonstrate the effectiveness of our approach on datasets such as Mip-NeRF360, Tanks and Temples, and Deep-Blending. Our method scores higher on photorealism metrics compared to other mesh extracting rendering methods without compromising mesh quality.

Problem

• 3D Gaussian Splattering
• Accuracy Improvement
• Visual Realism

Innovation

• 3D technology
• line direction information
• Gaussian splatting