Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learning

2026-02-07

AI Summary

This work addresses a key limitation in offline reinforcement learning: behavior cloning (BC) regularization often constrains policy performance by blindly imitating suboptimal actions, particularly suppressing exploration of high-value actions in later training stages. To mitigate this issue, the authors propose Proximal Action Replacement (PAR), a plug-and-play data augmentation mechanism within the Actor-Critic framework. PAR progressively replaces low-value actions in the replay buffer with high-value actions generated by a stabilized policy. This approach effectively reduces overreliance on suboptimal demonstrations, is compatible with various BC-regularized algorithms, and consistently yields significant performance gains across multiple offline RL benchmarks. Notably, when integrated with TD3+BC, PAR achieves performance approaching state-of-the-art levels.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Offline reinforcement learning (RL) optimizes policies from a previously collected static dataset and is an important branch of RL. A popular and promising approach is to regularize actor-critic methods with behavior cloning (BC), which yields realistic policies and mitigates bias from out-of-distribution actions, but can impose an often-overlooked performance ceiling: when dataset actions are suboptimal, indiscriminate imitation structurally prevents the actor from fully exploiting high-value regions suggested by the critic, especially in later training when imitation is already dominant. We formally analyzed this limitation by investigating convergence properties of BC-regularized actor-critic optimization and verified it on a controlled continuous bandit task. To break this ceiling, we propose proximal action replacement (PAR), a plug-and-play training sample replacer that progressively replaces low-value actions with high-value actions generated by a stable actor, broadening the action exploration space while reducing the impact of low-value data. PAR is compatible with multiple BC regularization paradigms. Extensive experiments across offline RL benchmarks show that PAR consistently improves performance and approaches state-of-the-art when combined with the basic TD3+BC.

Problem

• offline reinforcement learning
• behavior cloning
• actor-critic
• performance ceiling
• suboptimal actions

Innovation

• Proximal Action Replacement
• Offline Reinforcement Learning
• Behavior Cloning
• Actor-Critic
• Action Replacement