Offline-Online Reinforcement Learning for Linear Mixture MDPs

2026-04-13

AI Summary

This work addresses the challenge of integrating offline and online reinforcement learning under environmental shifts by proposing an adaptive algorithm within the linear mixture MDP framework. The algorithm employs a confidence interval–guided adaptive weighting mechanism to dynamically determine whether to leverage offline data that may originate from a different environment. Theoretical analysis establishes, for the first time, conditions under which offline data remains beneficial despite distributional shifts and derives matching upper and lower regret bounds. These results demonstrate that the algorithm strictly outperforms purely online methods when offline data provides sufficient coverage or environmental shifts are small, while automatically reverting to online learning to ensure robust performance otherwise. Empirical experiments validate the theoretical guarantees.

Technology Category

• Machine Learning: Online Learning & Bandits
• Multiagent Systems: Adversarial Agents
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

We study offline-online reinforcement learning in linear mixture Markov decision processes (MDPs) under environment shift. In the offline phase, data are collected by an unknown behavior policy and may come from a mismatched environment, while in the online phase the learner interacts with the target environment. We propose an algorithm that adaptively leverages offline data. When the offline data are informative, either due to sufficient coverage or small environment shift, the algorithm provably improves over purely online learning. When the offline data are uninformative, it safely ignores them and matches the online-only performance. We establish regret upper bounds that explicitly characterize when offline data are beneficial, together with nearly matching lower bounds. Numerical experiments further corroborate our theoretical findings.

Problem

• offline-online reinforcement learning
• linear mixture MDPs
• environment shift
• offline data utilization

Innovation

• offline-online reinforcement learning
• linear mixture MDPs
• environment shift
• adaptive data utilization
• regret bounds