Adaptive Bregman Proximal Stochastic Gradient with a Stabilized Barzilai--Borwein Step Size

2026-08-12

AI Summary

This work addresses the challenges of stepsize sensitivity, unstable curvature estimation, and high computational cost of line search in composite stochastic optimization over non-Euclidean geometries. The authors propose Ada-BPSG, a novel method that integrates a stabilized Barzilai–Borwein (BB) stepsize with Bregman proximal stochastic gradient, incorporating SAGA variance reduction, weighted median aggregation of secant information, and explicit stepsize constraints to enable adaptive stepsize selection without line search. The approach establishes a direct analytical framework linking relative smoothness to convergence, guaranteeing an $O(n/K)$ ergodic convergence rate under convexity, linear convergence with restarts under quadratic growth, and an $O(1/K)$ bound on proximal residuals in the non-convex setting. Experiments demonstrate superior performance in logistic regression and sparse non-negative matrix factorization, achieving lower objective values, robustness to initial stepsize choices, and elimination of line search overhead.

Abstract

Bregman proximal stochastic gradient (BPSG) methods bring variance-reduced composite optimization to objectives whose geometry is poorly captured by Euclidean smoothness. Their performance, however, remains sensitive to the step size: raw stochastic curvature estimates can fluctuate sharply, whereas line searches add repeated proximal evaluations. We introduce Ada-BPSG, a line-search-free BPSG method that couples the SAGA gradient table with a stabilized Barzilai--Borwein (BB) candidate. A mediant aggregates incremental secant information so that nearly singular local ratios receive little weight, and an explicit safeguard translates the resulting curvature estimate into the bounded step-size sequence required for convergence. This design yields a direct analytical chain from relative smoothness and component-wise variance control to convergence in finite-dimensional normed spaces. We prove an $O(n/K)$ ergodic rate for convex objectives, a restarted linear rate under relative quadratic growth, and an $O(1/K)$ bound for a Bregman proximal residual in the nonconvex setting. On logistic regression and sparse nonnegative matrix factorization, Ada-BPSG combines low objective values with substantially less sensitivity to the initial step size than standard variance-reduced baselines, while avoiding line search.

Problem

• step size sensitivity
• Bregman proximal stochastic gradient
• variance-reduced optimization
• non-Euclidean geometry
• stochastic curvature estimation

Innovation

• Bregman proximal gradient
• stabilized Barzilai–Borwein step size
• variance reduction
• adaptive step size
• relative smoothness