Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluation

2026-08-25

AI Summary

本文针对强化学习中策略评估的高方差问题，提出了一种双循环梯度算法来学习对转移不确定性鲁棒的行为策略。

Abstract

In reinforcement learning policy evaluation, classic on-policy methods often suffer from high variance when estimating policy performance. To mitigate this issue, behavior policy search has been proposed to learn data-collecting policies tailored to reduce online evaluation variance. However, these approaches do not account for uncertainties in the transition functions. In practice, simulator transitions often differ from the real world due to modeling errors or approximation limitations. As a result, behavior policies trained in simulation may still yield high variance when deployed in real environments, leading to costly reliance on real-world evaluation samples. In this work, we propose a double-loop gradient-based algorithm for learning behavior policies that are both efficient and robust to transition uncertainty. Theoretically, we derive novel transition-variance gradient expressions and establish global convergence guarantees for the algorithm. Numerically, we demonstrate that our method is less sensitive to transition perturbations than existing approaches, providing supportive evidence for its practical utility.

Problem

• reinforcement learning
• policy evaluation
• high variance
• transition uncertainty
• behavior policy

Innovation

• double-loop gradient-based algorithm
• transition uncertainty
• variance reduction
• global convergence