Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamics

2024-02-17

International Conference on Machine Learning

AI Summary

In non-stationary environments with limited offline data, existing methods conflate dynamics shifts with policy-induced distributional shifts and suffer from spurious context associations. To address this, we propose DORA—the first offline reinforcement learning framework to apply the information bottleneck principle for disentangling dynamics encoding from behavioral policy influence. DORA achieves unbiased dynamics representation through four key components: (i) mutual information upper-bound optimization, (ii) dynamics-aware context encoding, (iii) trajectory reweighting, and (iv) contrastive regularization. Evaluated on six MuJoCo non-stationary tasks, DORA significantly accelerates online adaptation and improves final performance: dynamics encoding accuracy increases by 32%, and average task return surpasses state-of-the-art baselines by 27%.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Developing policies that can adjust to non-stationary environments is essential for real-world reinforcement learning applications. However, learning such adaptable policies in offline settings, with only a limited set of pre-collected trajectories, presents significant challenges. A key difficulty arises because the limited offline data makes it hard for the context encoder to differentiate between changes in the environment dynamics and shifts in the behavior policy, often leading to context misassociations. To address this issue, we introduce a novel approach called Debiased Offline Representation for fast online Adaptation (DORA). DORA incorporates an information bottleneck principle that maximizes mutual information between the dynamics encoding and the environmental data, while minimizing mutual information between the dynamics encoding and the actions of the behavior policy. We present a practical implementation of DORA, leveraging tractable bounds of the information bottleneck principle. Our experimental evaluation across six benchmark MuJoCo tasks with variable parameters demonstrates that DORA not only achieves a more precise dynamics encoding but also significantly outperforms existing baselines in terms of performance.

Problem

• Adapt policies to non-stationary environments offline
• Differentiate environment dynamics from behavior policy shifts
• Improve dynamics encoding for fast online adaptation

Innovation

• Debiased Offline Representation for fast adaptation
• Information bottleneck principle for dynamics encoding
• Mutual information optimization between dynamics and data