Fast Convergence of Policy Regret in Learning Stochastic Optimal Control

2026-05-25

AI Summary

This work addresses value-based policy learning in operational settings characterized by limited data and a continuous, high-dimensional state-action space, achieving rapid regret convergence through greedy policies induced by Q*-function estimates. The core contribution lies in identifying three geometric structures governing convergence rates in continuous action spaces: the growth exponent \( p \), the boundary quality exponent \( m \), and the action regularity exponent \( q \). It is rigorously shown that when \( q > 0 \), policy regret converges faster than \( n^{-1/2} \). By integrating minimax analysis with Q* estimation, the theoretical results are validated in practical scenarios such as dynamic inventory management and service allocation, establishing minimax-optimal regret convergence rates and providing a solid theoretical foundation for efficient learning in high-dimensional continuous decision-making problems.

Abstract

Policy learning in modern operations environments faces a fundamental tension between limited operational data and the large, often continuous, state and action spaces over which good decisions must be identified and deployed. We study value-based policy learning in stochastic optimal control: a greedy policy induced by an estimate of the optimal action-value function $Q^*$ is deployed, and its performance is measured by regret. The empirical success of this approach calls for statistical insight into the structures that enable fast regret convergence. We show that, in continuous action spaces, fast policy learning is induced by three geometric structures: a growth exponent $p$, which quantifies how quickly $Q^*$ separates suboptimal actions from its maximizers; a margin-mass exponent $m$, which controls how much deployment mass lies on states with weak growth; and an action-wise regularity exponent $q$, which measures the smoothness of the $Q^*$-estimation error across actions. Given a $n^{-1/2}$-accurate estimator of $Q^*$, we show that the minimax-optimal policy regret convergence rate is \[ \widetildeΘ\left( n^{-\min\left\{\frac{p}{2(p-q)},\frac{m+1}{2m}\right\}} \right), \] up to a logarithmic factor at the boundary between the two regimes. The exponent $q$ is crucial: $q>0$ yields faster-than-$n^{-1/2}$ regret. This regime is natural in operations applications. In particular, we verify $q>0$ under mild regularity conditions in dynamic inventory control and service allocation examples, while the mechanism underlying this fast rate regime extends beyond these settings.

Problem

• policy regret
• stochastic optimal control
• continuous action spaces
• value-based policy learning
• fast convergence

Innovation

• policy regret
• stochastic optimal control
• fast convergence
• geometric structures
• action-value function