Actor-Critic with Active Importance Sampling

2026-05-07

AI Summary

This work proposes Active Importance Sampling Actor-Critic (AISAC), a novel algorithm addressing the high variance inherent in policy gradient methods, which often leads to inefficient learning and unstable training. AISAC treats the behavior policy as a learnable component and dynamically adapts its distribution via importance sampling to align with the target policy gradient direction, thereby minimizing estimator variance while preserving unbiasedness. Leveraging a Gaussian behavior policy optimized through cross-entropy minimization, the method enables efficient policy updates and value estimation in continuous control tasks. Empirical results demonstrate that AISAC significantly improves learning speed, sample efficiency, and training stability on benchmarks such as Inverted Pendulum and Half Cheetah, while exhibiting strong robustness to hyperparameter variations.

Abstract

This paper introduces the Active-Importance-Sampling Actor-Critic (AISAC) algorithm, an extension of the Actor-Critic framework for reducing variance in policy gradient estimation. AISAC optimizes the behavior policy to minimize gradient variance while preserving unbiased gradient estimates. Using importance sampling principles, the algorithm adapts the behavior policy toward efficient data collection distributions aligned with target policy gradients. For continuous action spaces, AISAC employs Gaussian behavior policies optimized through cross-entropy minimization. We provide theoretical analysis demonstrating variance reduction and unbiasedness. Experiments on Inverted Pendulum and Half Cheetah tasks show improved learning speed, sample efficiency, and training stability compared to standard Actor-Critic methods. Results indicate that optimizing the behavior policy improves both target policy updates and critic estimation accuracy across different hyperparameter settings. AISAC accelerates convergence and stabilizes reinforcement learning training, making it promising for real-world applications. Future work includes integration with advanced algorithms such as Soft Actor-Critic and TD3 for more complex environments.

Problem

• policy gradient variance
• Actor-Critic
• importance sampling
• sample efficiency
• training stability

Innovation

• Actor-Critic
• Importance Sampling
• Variance Reduction
• Behavior Policy Optimization
• Policy Gradient