Concentration of General Stochastic Approximation Under Heavy-Tailed Markovian Noise

2026-05-20

AI Summary

This work investigates the concentration of iteration errors in stochastic approximation algorithms driven by heavy-tailed Markov noise, covering both expansive and non-expansive operator settings. Under a framework involving a finite-state Markov component and martingale difference noise, the authors construct a novel Lyapunov function via the moment-generating function of the solution to the Poisson equation, complemented by auxiliary projection and black-box truncation techniques to reduce unbounded noise to a bounded setting. The study provides the first systematic characterization of the fine structure of error tails: under bounded noise, tails can be sub-Gaussian, sub-Weibull, or intermediate between Pareto and Weibull; under unbounded noise, if the operator is almost surely non-expansive, the error tail is at most three times heavier than that of the noise, whereas if the operator is expansive with positive probability, significantly heavier tails may arise, with sharp worst-case examples demonstrating the tightness of these bounds.

Abstract

We establish maximal concentration bounds for the iterates generated by stochastic approximation algorithms with general step sizes, where the noise has a finite-state Markovian component plus a Martingale-difference component. When the Martingale-difference noise is bounded, we show that the tail of the error can be sub-Gaussian, sub-Weibull, or something lighter than any Pareto but heavier than any Weibull, depending on the step size sequence and on whether the random operator is almost surely contractive, almost surely non-expansive, or expansive with positive probability. Our analysis relies on a novel Lyapunov function involving the moment-generating function of the solution to a Poisson equation, together with an auxiliary projected algorithm. We complement the upper bounds with worst-case examples showing that qualitatively sharper bounds are impossible. We further study the case of unbounded Martingale-difference noise when the average operator is contractive, and the step sizes are of order $1/k$. In this setting, we show that if the random operator is almost surely non-expansive, then the error tail is at most three times heavier than the noise tail, whereas if the random operator is expansive with positive probability, then the error may have substantially heavier tails. These results are obtained through a novel black-box truncation argument that reduces the unbounded-noise setting to the bounded-noise case.

Problem

• stochastic approximation
• heavy-tailed noise
• Markovian noise
• concentration bounds
• error tail behavior

Innovation

• stochastic approximation
• heavy-tailed noise
• Markovian noise
• Lyapunov function
• black-box truncation