Unified High-Probability Analysis of Stochastic Variance-Reduced Estimation

2026-05-14

AI Summary

This work addresses the limitations of existing stochastic variance reduction methods, whose analyses are often confined to expectation-level guarantees and tied to specific algorithms, thus failing to capture the structural trade-offs underlying reliability. The paper proposes a unified variance-reduced estimation framework that integrates memory retention, reset probability, and iterative moving correction, applicable in both Euclidean and non-Euclidean spaces. Its key contributions include the first dimension-free vector-valued Freedman inequality in smooth normed spaces, which—combined with martingale theory and mirror descent—enables a unified high-probability error analysis. Furthermore, the framework extends variance reduction to stochastic optimization with expectation constraints, achieving logarithmic confidence-dependent complexity for unconstrained mirror descent and improving the oracle complexity for constrained problems from ~O(ε⁻⁴) to ~O(ε⁻³).

Abstract

Stochastic estimators are fundamental to large-scale optimization, where population quantities must be inferred from noisy oracle observations. Although influential methods such as momentum, SPIDER, STORM, and PAGE have been highly successful, their analyses are largely estimator-specific and expectation-based, obscuring the structural tradeoffs that determine reliability. In this paper, we develop a unified framework for stochastic variance-reduced estimation based on a recursion with three components: memory retention, reset probability, and a correction term for iterate movement. This framework recovers several classical estimators, motivates new second-order variants, and yields a bias-variance decomposition of estimation error. Our main result is a unified high-probability bound proved using a new dimension-free vector-valued Freedman inequality, valid for smooth normed spaces involving random sums of vector martingales. The result applies in both Euclidean and non-Euclidean settings, including the analysis of mirror-descent-based methods in Banach spaces. As applications, we obtain high-probability oracle complexities for unconstrained optimization with mirror descent, establishing the logarithmic dependence on the confidence level. We also derive the first $\tilde{\mathcal{O}}(\varepsilon^{-3})$ oracle-complexity bounds for stochastic optimization with expectation constraints, improving upon the existing $\tilde{\mathcal{O}}(\varepsilon^{-4})$ complexity by leveraging variance-reduced estimation for the first time in this setting.

Problem

• stochastic variance-reduced estimation
• high-probability analysis
• unified framework
• oracle complexity
• expectation constraints

Innovation

• variance-reduced estimation
• high-probability bounds
• vector-valued Freedman inequality
• mirror descent
• stochastic optimization with expectation constraints