Faster Adaptive Optimization via Expected Gradient Outer Product Reparameterization

2025-02-03

AI Summary

Adaptive optimization algorithms (e.g., Adam, Adagrad) exhibit parameterization sensitivity, leading to unstable convergence. This work establishes a theoretical connection between the spectral decay property of the Expected Gradient Outer Product (EGOP) matrix and the algorithm’s basis-sensitivity. We propose, for the first time, an EGOP-based orthogonal reparameterization framework: by estimating the EGOP and applying an orthogonal transformation, it enhances rotational equivariance and mitigates step-size mismatch induced by anisotropic gradients. The method is plug-and-play, compatible with both stochastic and full-batch gradient approximations, and offers both theoretical interpretability and practical efficiency. Empirically, it accelerates convergence by 2–5× on convex and non-convex benchmarks. Theoretically, we prove that the adapted step sizes align with the local geometry—specifically, they become asymptotically matched to the eigenstructure of the Hessian or Fisher information matrix—thereby ensuring improved convergence guarantees for adaptive methods.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Adaptive optimization algorithms -- such as Adagrad, Adam, and their variants -- have found widespread use in machine learning, signal processing and many other settings. Several methods in this family are not rotationally equivariant, meaning that simple reparameterizations (i.e. change of basis) can drastically affect their convergence. However, their sensitivity to the choice of parameterization has not been systematically studied; it is not clear how to identify a"favorable"change of basis in which these methods perform best. In this paper we propose a reparameterization method and demonstrate both theoretically and empirically its potential to improve their convergence behavior. Our method is an orthonormal transformation based on the expected gradient outer product (EGOP) matrix, which can be approximated using either full-batch or stochastic gradient oracles. We show that for a broad class of functions, the sensitivity of adaptive algorithms to choice-of-basis is influenced by the decay of the EGOP matrix spectrum. We illustrate the potential impact of EGOP reparameterization by presenting empirical evidence and theoretical arguments that common machine learning tasks with"natural"data exhibit EGOP spectral decay.

Problem

• Adaptive Optimization Algorithms
• Parameter Representation
• Algorithm Performance Optimization

Innovation

• EGOP Matrix
• Parameter Representation Optimization
• Adaptive Optimization Algorithms