MaskGaussian: Adaptive 3D Gaussian Representation from Probabilistic Masks

2024-12-29

AI Summary

To address the excessive memory overhead in 3D Gaussian splatting caused by an overwhelming number of Gaussians, this paper proposes a probabilistic Gaussian modeling framework: each Gaussian is treated as a stochastic entity with a learnable existence probability; probabilistic forward rendering and gradient backpropagation are achieved via masked rasterization; and existence probabilities are adaptively updated in a gradient-driven manner to iteratively refine Gaussian importance and enable diversity-aware pruning. This work is the first to incorporate uncertainty modeling into Gaussian splatting optimization, overcoming the suboptimality inherent in conventional deterministic pruning strategies. Experiments demonstrate an average pruning rate exceeding 60%, with only a negligible PSNR degradation of 0.02, while achieving superior rendering quality using significantly fewer Gaussians.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Probabilistic Circuits and Graphical Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Data transparency and provenance

Abstract

While 3D Gaussian Splatting (3DGS) has demonstrated remarkable performance in novel view synthesis and real-time rendering, the high memory consumption due to the use of millions of Gaussians limits its practicality. To mitigate this issue, improvements have been made by pruning unnecessary Gaussians, either through a hand-crafted criterion or by using learned masks. However, these methods deterministically remove Gaussians based on a snapshot of the pruning moment, leading to sub-optimized reconstruction performance from a long-term perspective. To address this issue, we introduce MaskGaussian, which models Gaussians as probabilistic entities rather than permanently removing them, and utilize them according to their probability of existence. To achieve this, we propose a masked-rasterization technique that enables unused yet probabilistically existing Gaussians to receive gradients, allowing for dynamic assessment of their contribution to the evolving scene and adjustment of their probability of existence. Hence, the importance of Gaussians iteratively changes and the pruned Gaussians are selected diversely. Extensive experiments demonstrate the superiority of the proposed method in achieving better rendering quality with fewer Gaussians than previous pruning methods, pruning over 60% of Gaussians on average with only a 0.02 PSNR decline. Our code can be found at: https://github.com/kaikai23/MaskGaussian

Problem

• 3D Gaussian Splatting
• Storage Space Reduction
• Image Quality Preservation

Innovation

• MaskGaussian
• Adaptive Gaussian Modeling
• Storage Efficiency