Comparing BFGS and OGR for Second-Order Optimization

2025-12-07

AI Summary

Addressing the challenges of Hessian estimation—namely, high computational cost and numerical instability—in high-dimensional nonconvex optimization, this paper proposes Online Gradient Regression (OGR). OGR models the local second-order relationship between gradients and parameter displacements via exponential moving averages, enabling direct estimation of general (not necessarily positive-definite) Hessian matrices—thereby relaxing the positive-definiteness requirement inherent in BFGS-type methods. By integrating Sherman–Morrison updates with an online learning framework, OGR avoids explicit Hessian storage and inversion, significantly improving both computational efficiency and numerical robustness. Experiments on standard nonconvex benchmark functions demonstrate that OGR achieves faster convergence and lower final loss than BFGS, especially in high-dimensional settings. This work establishes a new, scalable, and robust paradigm for Hessian approximation in second-order optimization of neural networks.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Estimating the Hessian matrix, especially for neural network training, is a challenging problem due to high dimensionality and cost. In this work, we compare the classical Sherman-Morrison update used in the popular BFGS method (Broy-den-Fletcher-Goldfarb-Shanno), which maintains a positive definite Hessian approximation under a convexity assumption, with a novel approach called Online Gradient Regression (OGR). OGR performs regression of gradients against positions using an exponential moving average to estimate second derivatives online, without requiring Hessian inversion. Unlike BFGS, OGR allows estimation of a general (not necessarily positive definite) Hessian and can thus handle non-convex structures. We evaluate both methods across standard test functions and demonstrate that OGR achieves faster convergence and improved loss, particularly in non-convex settings.

Problem

• Compares BFGS and OGR for Hessian estimation in optimization
• Evaluates methods for handling non-convex structures in neural networks
• Assesses convergence speed and loss improvement in non-convex settings

Innovation

• OGR uses exponential moving average for Hessian estimation
• OGR avoids Hessian inversion unlike BFGS method
• OGR handles non-convex structures with general Hessian