Generalization in offline RL: The structure is more important than the amount of pessimism

2026-07-02

AI Summary

This work investigates the relationship between generalization and pessimism in offline reinforcement learning, arguing that generalization performance hinges not on the degree of pessimism per se, but on whether the pessimistic structure aligns with the symmetry of the optimal policy. To this end, the authors propose a symmetry-preserving pessimistic value function and introduce a consistency-based data augmentation strategy integrated into Implicit Q-Learning (IQL) and Conservative Q-Learning (CQL). Theoretical analysis is grounded in contextual Markov decision processes, and empirical evaluation in a rotationally symmetric Reacher environment demonstrates that the proposed approach substantially outperforms conventional offline data augmentation methods, achieving significantly improved generalization capabilities.

Abstract

While pessimism counteracts overestimation bias in offline reinforcement learning (RL), being overly conservative has been associated with hindering certain forms of generalization. However, in this paper we demonstrate that being overly pessimistic does not inherently prevent optimal generalization in contextual MDPs (CMDPs). Instead, we argue successful generalization depends not on the amount of pessimism, but whether the pessimistic structure respects the underlying symmetries of the optimal solution. We prove that a mildly pessimistic, non-symmetric value function can generalize worse than an overly pessimistic, symmetric one. In offline RL, the structure of the pessimism is determined by the structure of the dataset coverage. As such, enforcing a symmetric value function can be non-trivial, and might require techniques such as data augmentation (DA). Inspired by our theoretical results, we argue that DA can best be applied through a consistency loss during policy extraction, rather than the common practice of (regular) offline training on an augmented dataset. This is empirically validated using IQL and CQL on a rotationally symmetric reacher environment.

Problem

• offline reinforcement learning
• generalization
• pessimism
• symmetry
• contextual MDPs

Innovation

• offline reinforcement learning
• generalization
• symmetry
• pessimism structure
• data augmentation