🤖 AI Summary
This paper addresses the challenge of estimating policy gradients in partially observable Markov decision processes (POMDPs). We propose GPOMDP, a direct policy gradient estimation algorithm that operates on a single trajectory and involves only one tunable parameter. GPOMDP extends the REINFORCE framework to average-reward optimization in POMDPs without requiring a transition model or prior domain knowledge. Its gradient estimator admits an explicit bias–variance trade-off interpretation and can be integrated with conjugate gradient methods to accelerate convergence. We provide theoretical convergence guarantees for the algorithm under standard regularity conditions. Empirical evaluations demonstrate that GPOMDP robustly approximates the true policy gradient in complex POMDP environments and reliably converges to locally optimal policies maximizing the average reward.
📝 Abstract
This paper discusses theoretical and experimental aspects of gradient-based approaches to the direct optimization of policy performance in controlled POMDPs. We introduce GPOMDP, a REINFORCE-like algorithm for estimating an approximation to the gradient of the average reward as a function of the parameters of a stochastic policy. The algorithm’s chief advantages are that it requires only a single sample path of the underlying Markov chain, it uses only one free parameter 2 [0; 1), which has a natural interpretation in terms of bias-variance trade-off, and it requires no knowledge of the underlying state. We prove convergence of GPOMDP and show how the gradient estimates produced by GPOMDP can be used in a conjugate-gradient procedure to find local optima of the average reward.