3DGS-LM: Faster Gaussian-Splatting Optimization with Levenberg-Marquardt

2024-09-19

arXiv.org

AI Summary

Existing 3D Gaussian Splatting (3DGS) reconstruction relies on the ADAM optimizer, requiring thousands of iterations and approximately one hour of training time. Method: We propose the first Levenberg–Marquardt (LM)-based optimization framework tailored for 3DGS, featuring a GPU-efficient gradient caching data structure and custom CUDA kernels to accelerate Jacobian-vector products and enable weighted updates over image subsets. Our method tightly integrates differentiable rasterization with LM’s second-order optimization mechanism, yielding superior parameter update directions within a single forward–backward pass. Contribution/Results: Experiments demonstrate a 30% speedup over vanilla 3DGS without compromising PSNR or SSIM performance. Moreover, our approach is orthogonal to existing acceleration techniques—such as sparsification and pruning—enabling synergistic improvements. This work establishes a new paradigm for real-time 3D reconstruction.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Optimization
• Computer Vision: 3D Computer Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We present 3DGS-LM, a new method that accelerates the reconstruction of 3D Gaussian Splatting (3DGS) by replacing its ADAM optimizer with a tailored Levenberg-Marquardt (LM). Existing methods reduce the optimization time by decreasing the number of Gaussians or by improving the implementation of the differentiable rasterizer. However, they still rely on the ADAM optimizer to fit Gaussian parameters of a scene in thousands of iterations, which can take up to an hour. To this end, we change the optimizer to LM that runs in conjunction with the 3DGS differentiable rasterizer. For efficient GPU parallization, we propose a caching data structure for intermediate gradients that allows us to efficiently calculate Jacobian-vector products in custom CUDA kernels. In every LM iteration, we calculate update directions from multiple image subsets using these kernels and combine them in a weighted mean. Overall, our method is 30% faster than the original 3DGS while obtaining the same reconstruction quality. Our optimization is also agnostic to other methods that acclerate 3DGS, thus enabling even faster speedups compared to vanilla 3DGS.

Problem

• Accelerating 3D Gaussian Splatting reconstruction optimization
• Replacing ADAM optimizer with tailored Levenberg-Marquardt method
• Reducing optimization time while maintaining reconstruction quality

Innovation

• Replaces ADAM optimizer with Levenberg-Marquardt
• Uses caching structure for GPU gradient computation
• Combines updates from multiple image subsets