SVD-Preconditioned Gradient Descent Method for Solving Nonlinear Least Squares Problems

2026-02-07

AI Summary

This work proposes a novel algorithm for nonlinear least squares problems that integrates singular value decomposition (SVD)-based preconditioned gradient directions with the Adam adaptive learning rate mechanism. Addressing the limitations of conventional optimization methods—namely slow convergence and low accuracy—the approach uniquely combines SVD preconditioning with Adam, achieving local linear convergence while theoretically guaranteeing global convergence. Experimental evaluations on function approximation, partial differential equation solving, and CIFAR-10 image classification demonstrate that the proposed method significantly outperforms standard Adam in both convergence speed and solution accuracy, thereby confirming its effectiveness and strong generalization capability across diverse tasks.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

This paper introduces a novel optimization algorithm designed for nonlinear least-squares problems. The method is derived by preconditioning the gradient descent direction using the Singular Value Decomposition (SVD) of the Jacobian. This SVD-based preconditioner is then integrated with the first- and second-moment adaptive learning rate mechanism of the Adam optimizer. We establish the local linear convergence of the proposed method under standard regularity assumptions and prove global convergence for a modified version of the algorithm under suitable conditions. The effectiveness of the approach is demonstrated experimentally across a range of tasks, including function approximation, partial differential equation (PDE) solving, and image classification on the CIFAR-10 dataset. Results show that the proposed method consistently outperforms standard Adam, achieving faster convergence and lower error in both regression and classification settings.

Problem

• nonlinear least squares
• optimization
• convergence
• preconditioning
• gradient descent

Innovation

• SVD preconditioning
• nonlinear least squares
• adaptive optimization
• gradient descent
• convergence analysis