🤖 AI Summary
This work addresses the challenge of optimizing complex loss functions—such as Kullback–Leibler divergence or PDE residuals—over nonlinear manifolds like neural or tensor networks, where conventional and natural gradient descent often converge to suboptimal local minima with inefficient update directions. The paper introduces, for the first time, a momentum-augmented variant of natural gradient descent that integrates Heavy-Ball and Nesterov-type inertial mechanisms into a manifold-aware optimization framework. By leveraging tangent-space projections and Gram matrix preconditioning, the method achieves momentum-driven, locally optimal updates directly in function space. Empirical results demonstrate substantial improvements in convergence behavior for tasks including density estimation and physics-informed learning, effectively mitigating poor local minima and enhancing the quality of optimization trajectories.
📝 Abstract
We consider the problem of approximating a function by an element of a nonlinear manifold which admits a differentiable parametrization, typical examples being neural networks with differentiable activation functions or tensor networks. Natural gradient descent (NGD) for the optimization of a loss function can be seen as a preconditioned gradient descent where updates in the parameter space are driven by a functional perspective. In a spirit similar to Newton's method, a NGD step uses, instead of the Hessian, the Gram matrix of the generating system of the tangent space to the approximation manifold at the current iterate, with respect to a suitable metric. This corresponds to a locally optimal update in function space, following a projected gradient onto the tangent space to the manifold. Still, both gradient and natural gradient descent methods get stuck in local minima. Furthermore, when the model class is a nonlinear manifold or the loss function is not ideally conditioned (e.g., the KL-divergence for density estimation, or a norm of the residual of a partial differential equation in physics informed learning), even the natural gradient might yield non-optimal directions at each step. This work introduces a natural version of classical inertial dynamic methods like Heavy-Ball or Nesterov and show how it can improve the learning process when working with nonlinear model classes.