Mani-GS: Gaussian Splatting Manipulation with Triangular Mesh

2024-05-28

arXiv.org

AI Summary

This work addresses the limited editability of 3D Gaussian Splatting (3DGS). We propose the first general-purpose, triangle-mesh-driven editing framework for 3DGS. Methodologically, we introduce a shape-aware Gaussian-vertex adaptive binding mechanism, enabling large deformations, localized edits, and soft-body simulation; Gaussian positions and covariances are updated in real time via mesh deformation—eliminating the need for task-specific algorithms and ensuring robustness to imprecise input meshes. Our key contributions are: (i) the first integration of explicit triangle meshes into the 3DGS editing paradigm, unifying high-fidelity rendering with strong controllability; (ii) novel editing capabilities while preserving view synthesis quality on par with the original 3DGS; and (iii) significantly faster training and inference compared to NeRF-based methods, with consistent superiority across diverse editing tasks over existing baselines.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Learning with Manifolds

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Neural 3D representations such as Neural Radiance Fields (NeRF), excel at producing photo-realistic rendering results but lack the flexibility for manipulation and editing which is crucial for content creation. Previous works have attempted to address this issue by deforming a NeRF in canonical space or manipulating the radiance field based on an explicit mesh. However, manipulating NeRF is not highly controllable and requires a long training and inference time. With the emergence of 3D Gaussian Splatting (3DGS), extremely high-fidelity novel view synthesis can be achieved using an explicit point-based 3D representation with much faster training and rendering speed. However, there is still a lack of effective means to manipulate 3DGS freely while maintaining rendering quality. In this work, we aim to tackle the challenge of achieving manipulable photo-realistic rendering. We propose to utilize a triangular mesh to manipulate 3DGS directly with self-adaptation. This approach reduces the need to design various algorithms for different types of Gaussian manipulation. By utilizing a triangle shape-aware Gaussian binding and adapting method, we can achieve 3DGS manipulation and preserve high-fidelity rendering after manipulation. Our approach is capable of handling large deformations, local manipulations, and soft body simulations while keeping high-quality rendering. Furthermore, we demonstrate that our method is also effective with inaccurate meshes extracted from 3DGS. Experiments conducted demonstrate the effectiveness of our method and its superiority over baseline approaches.

Problem

• Enabling flexible manipulation of 3D Gaussian Splatting (3DGS) while preserving rendering quality
• Reducing complexity by using triangular mesh for adaptive 3DGS control
• Achieving high-fidelity rendering after large deformations or local edits

Innovation

• Uses triangular mesh for 3DGS manipulation
• Self-adapting Gaussian binding preserves rendering
• Handles large deformations and local edits