Problem
Research questions and friction points this paper is trying to address.
offline reinforcement learning
actor-critic methods
out-of-distribution actions
critic overestimation
policy improvement
Decoupling Policy Extraction for Offline Reinforcement Learning
🤖 AI Summary
本文针对离线强化学习中演员与评论家联合训练导致的问题，提出一种解耦策略提取方法，通过单独训练演员模仿行为分布，并在推理时用独立学习的评论家对多个行动建议重排序来改善策略。
📝 Abstract
Offline RL methods commonly jointly train the actor and critic, where the critic is used to guide the actor toward higher-value actions. This coupled learning process is well motivated in online RL, where an improved actor collects new data that can further update the actor and the critic. However, training data remains fixed in offline RL, making actor-side policy improvement unable to generate new data to validate or correct the critic. Moreover, retaining this coupled paradigm leads to two related challenges. Firstly, actor updates can drift toward high-valued but potentially out-of-distribution (OOD) actions and amplify critic overestimation. Secondly, conservative value estimation or behavior-cloning regularization creates a difficult trade-off between suppressing OOD actions and selecting high-value actions within the data-supported region. Motivated by this observation, we revisit the conventional offline RL paradigm and propose decoupling policy improvement from actor training. Specifically, we train the actor solely to model the behavior distribution and perform policy improvement at inference time by reranking multiple actor-generated proposals with a separately learned critic. We refer to this paradigm as the decoupled policy extraction paradigm. Under such paradigm, the actor provides behavior-supported action candidates, while the critic performs value-based selection within this candidate set. Extensive experiments show that the decoupled policy extraction paradigm outperforms both behavior cloning and jointly learned offline RL methods, while remaining effective even with a naive Q-learning critic.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Decoupling Policy Extraction
Offline Reinforcement Learning
Behavior Distribution Modeling
🔎 Similar Papers
No similar papers found.
