Sample Complexity of Policy Gradient for Log-Growth Control

2026-05-26

AI Summary

This work addresses the problem of learning optimal feedback gains for scalar linear systems driven by multiplicative noise, with the objective of minimizing the maximal Lyapunov exponent of the closed-loop system. To overcome the challenge that the gradient estimator’s variance diverges due to a singularity in the objective function at the optimum, the authors exploit the symmetric structure of a “cusp barrier” and employ reflected paired observations to cancel the divergent terms in the gradient estimate, thereby simultaneously controlling curvature, variance, and density estimation bias. Leveraging a projected mini-batch policy gradient algorithm together with a closed-form one-step transition gradient oracle, the method achieves a sample complexity of Õ(1/η) when the noise density is known. When the density belongs to a C^s class (s ≥ 2) and must be estimated, the sample complexity becomes Õ(η^{-(2s+1)/(2s)}).

Abstract

We study the sample complexity of policy gradient for log-growth control -- the problem of learning, from observed state transitions, a feedback gain that optimally stabilizes a scalar linear system driven through a multiplicative-noise actuation channel. The objective $J(K) = \mathbb{E}[\log|1+BK|]$ is the top Lyapunov exponent of the closed loop. This problem carries a structural difficulty we call the cusp obstruction: the optimal gain $K^*$ always places the noise singularity $b_{\rm sing}(K) = -1/K$ in the interior of the support. At this singular optimum the policy gradient exists only as a Cauchy principal value, not as a Lebesgue integral, and the natural single-sample gradient estimator has infinite variance. Standard first-order stochastic-optimization analysis is thus inapplicable at the optimum, and merely smoothing the objective does not resolve the difficulty. The obstruction, however, has an exploitable symmetry: the Cauchy kernel is an odd function of the displacement from the moving pole, so pairing each observation with its reflection through the pole cancels the divergent part. This one cancellation simultaneously controls the population curvature, the gradient-estimator variance, and the bias incurred when the noise density is estimated. Combining these bounds with a closed-form single-transition gradient oracle, we prove that projected mini-batch policy gradient, initialized in any compact subset of the stabilizing region, attains total sample complexity $\tilde{O}(1/η)$ when the noise density is known and $\tilde{O}(η^{-(2s+1)/(2s)})$ when it must be estimated, for $C^s$ noise densities with $s \geq 2$.

Problem

• sample complexity
• policy gradient
• log-growth control
• multiplicative noise
• cusp obstruction

Innovation

• cusp obstruction
• policy gradient
• sample complexity
• Cauchy principal value
• multiplicative noise