Pessimistic Auxiliary Policy for Offline Reinforcement Learning

2026-02-27

AI Summary

Offline reinforcement learning is prone to value overestimation and error accumulation due to out-of-distribution actions. To address this issue, this work proposes a pessimistic auxiliary policy based on the lower confidence bound of the Q-function, which samples actions within the neighborhood of the current policy that exhibit both high estimated returns and low uncertainty. This approach effectively mitigates value overestimation and prevents error propagation during training. Empirical evaluations demonstrate that the proposed method significantly outperforms existing algorithms across multiple standard offline reinforcement learning benchmarks, highlighting its effectiveness and broad applicability.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Offline reinforcement learning aims to learn an agent from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, inevitable access to out-ofdistribution actions during the learning process introduces approximation errors, causing the error accumulation and considerable overestimation. In this paper, we construct a new pessimistic auxiliary policy for sampling reliable actions. Specifically, we develop a pessimistic auxiliary strategy by maximizing the lower confidence bound of the Q-function. The pessimistic auxiliary strategy exhibits a relatively high value and low uncertainty in the vicinity of the learned policy, avoiding the learned policy sampling high-value actions with potentially high errors during the learning process. Less approximation error introduced by sampled action from pessimistic auxiliary strategy leads to the alleviation of error accumulation. Extensive experiments on offline reinforcement learning benchmarks reveal that utilizing the pessimistic auxiliary strategy can effectively improve the efficacy of other offline RL approaches.

Problem

• offline reinforcement learning
• out-of-distribution actions
• approximation error
• error accumulation
• overestimation

Innovation

• pessimistic auxiliary policy
• offline reinforcement learning
• lower confidence bound
• error accumulation
• out-of-distribution actions