SILAGE: Memory-Efficient, Full-Gradient-Free Nonconvex Optimization for Nested Finite Sums

2026-06-14

AI Summary

This work addresses the high computational and memory costs incurred by existing variance-reduction methods in large-scale empirical risk minimization with a nested finite-sum structure, which typically require expensive full-gradient evaluations or substantial memory. The authors propose SILAGE, an algorithm that exploits the nested grouping of data to compute only a single local group gradient per iteration, thereby eliminating the need for periodic global full-gradient updates and reducing memory complexity to O(n). SILAGE is the first method to achieve low memory complexity without global gradient computations, and it adaptively leverages the geometric structure of the data through inter-group (δ₁) and intra-group (δ₂) function similarity, circumventing the conventional reliance on worst-case Lipschitz constants. Theoretical analysis demonstrates that SILAGE improves upon the best-known complexity bounds of existing algorithms in several practical settings, significantly lowering both computational and memory overhead.

Abstract

Empirical risk minimization on massive datasets naturally exhibits a nested double finite-sum structure, where $N=nm$ total samples are logically or physically partitioned into $n$ blocks of size $m$ (e.g., in pooled data silos, out-of-core learning, or deliberate stratification). While variance-reduced methods achieve optimal oracle complexities for nonconvex objectives, they suffer from severe scaling bottlenecks in this centralized regime. Recursive estimators, such as PAGE, require periodic global full-gradient refreshes over all $nm$ samples, which are computationally expensive. Conversely, single-loop methods, such as SILVER, avoid such refreshes but require an impractical $\mathcal{O}(nm)$ memory footprint to store a control variate for every sample. In this paper, we propose SILAGE, a variance-reduced algorithm that addresses this trade-off. By actively exploiting the double-sum structure, SILAGE eliminates periodic global full-gradient refreshes over all $nm$ components (evaluating at most one local group gradient per iteration) while requiring only $\mathcal{O}(n)$ memory. Furthermore, we provide a tight convergence analysis that avoids pessimistic worst-case Lipschitz constants. Instead, SILAGE's complexity natively adapts to the underlying data geometry via nested functional similarities: across-group ($δ_1$) and within-group ($δ_2$) heterogeneity. Our results improve existing state-of-the-art bounds in several practically relevant regimes.

Problem

• nested finite sums
• nonconvex optimization
• variance reduction
• memory efficiency
• empirical risk minimization

Innovation

• nested finite sums
• variance reduction
• memory efficiency
• gradient-free optimization
• adaptive convergence