Learn2Splat: Extending the Horizon of Learned 3DGS Optimization

2026-05-15

AI Summary

This work addresses the slow convergence and inefficiency of conventional 3D Gaussian Splatting (3DGS) optimizers, which often neglect scene structure and spatial relationships. The authors propose a learnable optimizer tailored for 3DGS, integrating a meta-learning framework with a gradient-scale-aware architecture to enable stable, long-horizon optimization without manual learning rate scheduling. By incorporating a checkpoint buffer and an optimizer unrolling strategy, the method mitigates performance degradation over extended optimization horizons—without relying on auxiliary mechanisms—for the first time. The approach is effective under both sparse and dense input views, significantly improving early-stage novel view synthesis quality while maintaining stability during prolonged optimization, and demonstrates strong zero-shot generalization capabilities.

Abstract

3D Gaussian Splatting (3DGS) optimization is most commonly performed using standard optimizers (Adam, SGD). While stable across diverse scenes, standard optimizers are general-purpose and not tailored to the structure of the problem. In particular, they produce independent parameter updates that do not capture the structural and spatial relationships within a scene, leading to inefficient optimization and slow convergence. Recent works introduced learned optimizers that predict correlated updates informed by inter-parameter and inter-Gaussian dependencies. However, these methods are trained for a fixed number of optimization iterations and rely on manually scheduled learning rates to avoid degradation. In this paper, we introduce a learned optimizer for 3DGS that avoids degradation over extended optimization horizons without auxiliary mechanisms. To enable this, we propose a meta-learning scheme that extends the optimization horizon via a checkpoint buffer and an optimizer rollout strategy, combined with an architecture that encodes gradient scale information in its latent states. Results show improved early novel view synthesis quality while remaining stable over long horizons, with zero-shot generalization to unseen reconstruction settings. To support our findings, we introduce the first unified framework for training and evaluating both learned and conventional optimizers across sparse and dense view settings. Code and models will be released publicly. Our project page is available at https://naamapearl.github.io/learn2splat .

Problem

• 3D Gaussian Splatting
• learned optimizer
• optimization horizon
• parameter dependencies
• convergence efficiency

Innovation

• learned optimizer
• 3D Gaussian Splatting
• meta-learning
• gradient scale encoding
• zero-shot generalization