Finite-Time Analysis of Projected Two-Time-Scale Stochastic Approximation

2026-03-31

AI Summary

This work investigates the finite-time convergence of projected linear two-timescale stochastic approximation algorithms. For the constant-stepsize variant combined with Polyak–Ruppert averaging, it establishes the first explicit mean-squared error bound, cleanly decomposing it into an approximation error dictated by the constraint subspace and a statistical error that decays at a sublinear rate. This decomposition hinges on a restricted stability margin and a coupling invertibility condition, effectively disentangling the influence of subspace selection from that of the averaging window. The theoretical findings are validated through experiments on both synthetic data and reinforcement learning tasks, confirming the accuracy of the error decomposition and the algorithm’s superior performance.

Abstract

We study the finite-time convergence of projected linear two-time-scale stochastic approximation with constant step sizes and Polyak--Ruppert averaging. We establish an explicit mean-square error bound, decomposing it into two interpretable components, an approximation error determined by the constrained subspace and a statistical error decaying at a sublinear rate, with constants expressed through restricted stability margins and a coupling invertibility condition. These constants cleanly separate the effect of subspace choice (approximation errors) from the effect of the averaging horizon (statistical errors). We illustrate our theoretical results through a number of numerical experiments on both synthetic and reinforcement learning problems.

Problem

• two-time-scale stochastic approximation
• finite-time convergence
• mean-square error
• constrained subspace
• Polyak-Ruppert averaging

Innovation

• two-time-scale stochastic approximation
• finite-time analysis
• Polyak–Ruppert averaging
• mean-square error bound
• constrained subspace