Neural Surface Priors for Editable Gaussian Splatting

2024-11-27

arXiv.org

AI Summary

To address the limited flexibility of existing 3D scene editing methods, this paper proposes an editable reconstruction framework that synergistically integrates neural implicit geometric priors with 3D Gaussian splatting. Methodologically, it introduces the first SDF-guided Gaussian optimization, leveraging a lightweight triangular mesh proxy to enable topology-aware, seamless edit propagation. Joint mesh-Gaussian optimization coupled with differentiable rendering supports diverse interactive operations—including deformation, part replacement, and topological modification—while preserving geometric consistency. Extensive evaluation on multiple standard benchmarks demonstrates high-fidelity reconstruction and real-time editing capability. Our approach achieves significantly higher edit fidelity than state-of-the-art methods, effectively breaking the long-standing trade-off between edit controllability and geometric consistency inherent in conventional implicit or explicit representations.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In computer graphics and vision, recovering easily modifiable scene appearance from image data is crucial for applications such as content creation. We introduce a novel method that integrates 3D Gaussian Splatting with an implicit surface representation, enabling intuitive editing of recovered scenes through mesh manipulation. Starting with a set of input images and camera poses, our approach reconstructs the scene surface using a neural signed distance field. This neural surface acts as a geometric prior guiding the training of Gaussian Splatting components, ensuring their alignment with the scene geometry. To facilitate editing, we encode the visual and geometric information into a lightweight triangle soup proxy. Edits applied to the mesh extracted from the neural surface propagate seamlessly through this intermediate structure to update the recovered appearance. Unlike previous methods relying on the triangle soup proxy representation, our approach supports a wider range of modifications and fully leverages the mesh topology, enabling a more flexible and intuitive editing process. The complete source code for this project can be accessed at: https://github.com/WJakubowska/NeuralSurfacePriors.

Problem

• Enhance scene appearance editing
• Integrate Gaussian Splatting with surfaces
• Enable flexible mesh-based modifications

Innovation

• Integrates Gaussian Splatting with surfaces
• Uses neural signed distance fields
• Encodes data into triangle soup proxy