3D Gaussian Splatting as Markov Chain Monte Carlo

2024-04-15

Neural Information Processing Systems

AI Summary

Existing 3D Gaussian splatting methods rely on manual cloning/segmentation and sensitive initialization, resulting in unstable rendering quality and poor controllability over Gaussian count. This work reformulates the 3D Gaussian point set as a Monte Carlo Markov Chain (MCMC) sampling process of the scene’s physical representation. We replace heuristic Gaussian splitting/cloning with stochastic gradient Langevin dynamics (SGLD), and recast densification and pruning as deterministic MCMC state transitions. A probabilistic density relocation mechanism supplants cloning, while an unused-Gaussian regularizer enables flexible control over Gaussian count and robust initialization. Evaluated on standard benchmarks, our method achieves significantly improved rendering quality, reduces reliance on manual hyperparameter tuning, and unifies neural rendering with principled probabilistic modeling.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Probabilistic Circuits and Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

While 3D Gaussian Splatting has recently become popular for neural rendering, current methods rely on carefully engineered cloning and splitting strategies for placing Gaussians, which can lead to poor-quality renderings, and reliance on a good initialization. In this work, we rethink the set of 3D Gaussians as a random sample drawn from an underlying probability distribution describing the physical representation of the scene-in other words, Markov Chain Monte Carlo (MCMC) samples. Under this view, we show that the 3D Gaussian updates can be converted as Stochastic Gradient Langevin Dynamics (SGLD) updates by simply introducing noise. We then rewrite the densification and pruning strategies in 3D Gaussian Splatting as simply a deterministic state transition of MCMC samples, removing these heuristics from the framework. To do so, we revise the 'cloning' of Gaussians into a relocalization scheme that approximately preserves sample probability. To encourage efficient use of Gaussians, we introduce a regularizer that promotes the removal of unused Gaussians. On various standard evaluation scenes, we show that our method provides improved rendering quality, easy control over the number of Gaussians, and robustness to initialization.

Problem

• Improves 3D Gaussian Splatting rendering quality
• Converts Gaussian updates to SGLD updates
• Introduces regularizer for efficient Gaussian use

Innovation

• MCMC for 3D Gaussian updates
• SGLD with noise introduction
• Regularizer for Gaussian removal