🤖 AI Summary
This work addresses the challenge of learning memory-based policies for infinite-horizon partially observable Markov decision processes (POMDPs). We propose a scalable internal-state policy gradient method. Our approach unifies model-based optimization and model-free simulation training within a single framework. Technically, we introduce a novel gradient estimator and internal-state update mechanism that jointly enhance policy representational capacity and training stability. Additionally, we incorporate computationally efficient state compression and variance reduction techniques. Evaluated on large-scale POMDP benchmarks—including noisy robot navigation and cooperative multi-agent tasks—our method substantially outperforms existing memory-augmented policy approaches. It achieves superior convergence speed, policy performance, and generalization across diverse partially observable sequential decision-making scenarios. The proposed solution thus provides a practical, scalable reinforcement learning framework for complex POMDPs.
📝 Abstract
Policy-gradient methods have received increased attention recently as a mechanism for learning to act in partially observable environments. They have shown promise for problems admitting memoryless policies but have been less successful when memory is required. In this paper we develop several improved algorithms for learning policies with memory in an infinite-horizon setting -- directly when a known model of the environment is available, and via simulation otherwise. We compare these algorithms on some large POMDPs, including noisy robot navigation and multi-agent problems.