Time-Uniform Self-Normalized Concentration for Discounted Least Squares: Limits and Corrections

2026-08-20

AI Summary

研究解决了折扣最小二乘估计器在非平稳问题中的自归一化集中不等式的错误，并通过修正方法保证了其在固定时间的有效性。

Abstract

Self-normalized concentration inequalities are standard tools in bandit and reinforcement-learning analyses. A widely used weighted extension claims an analogous time-uniform guarantee for discounted least-squares estimators in non-stationary problems. A simple scalar Gaussian counterexample with a fixed parameter shows that the claimed bounded radius is crossed with probability one. For fixed discount and regularization parameters, we further show that, when $δ\leq1/2$ and $T/δ$ is sufficiently large, any deterministic anytime boundary valid uniformly over the stated conditionally sub-Gaussian model class must be at least of order $R\sqrt{\log(T/δ)}$ at some time by horizon $T$; for nondecreasing boundaries, this order is required at time $T$. We identify the proof error: different terminal times use different Gaussian mixing distributions, so the fixed-time mixtures do not form one supermartingale, and the stopping-time argument does not repair this failure. Finally, we show that the weighted inequality remains valid at each fixed deterministic time, give valid finite- and infinite-horizon corrections, and discuss consequences for downstream analyses.

Problem

• self-normalized concentration
• discounted least-squares
• non-stationary problems
• time-uniform guarantee
• sub-Gaussian model

Innovation

• self-normalized concentration
• discounted least squares
• supermartingale
• stopping-time argument
• finite- and infinite-horizon corrections