Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis

2025-05-19

AI Summary

This paper studies online reinforcement learning (RL) with offline data augmentation, aiming to synergistically leverage both offline datasets and online interaction to overcome the performance limitations of purely online or purely offline approaches. The authors propose the first unified hybrid algorithmic framework for this setting. They establish the first theoretical characterization revealing the heterogeneous requirements of suboptimality minimization and regret minimization on offline data coverage, and introduce a policy-dependent concentrability coefficient to quantify data quality. Building upon confidence-based online RL design and rigorous theoretical analysis, the method achieves dual optimality in linear contextual bandits and MDPs: the suboptimality gap attains $ ilde{mathcal{O}}ig(sqrt{1/(N_0/C(pi^* mid ho) + N_1)}ig)$, and the online regret acceleration ratio reaches $ ilde{mathcal{O}}ig(sqrt{N_1/(N_0/C(pi^- mid ho) + N_1)}ig)$, significantly improving over existing methods.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

This paper investigates a hybrid learning framework for reinforcement learning (RL) in which the agent can leverage both an offline dataset and online interactions to learn the optimal policy. We present a unified algorithm and analysis and show that augmenting confidence-based online RL algorithms with the offline dataset outperforms any pure online or offline algorithm alone and achieves state-of-the-art results under two learning metrics, i.e., sub-optimality gap and online learning regret. Specifically, we show that our algorithm achieves a sub-optimality gap $ ilde{O}(sqrt{1/(N_0/mathtt{C}(pi^*| ho)+N_1}) )$, where $mathtt{C}(pi^*| ho)$ is a new concentrability coefficient, $N_0$ and $N_1$ are the numbers of offline and online samples, respectively. For regret minimization, we show that it achieves a constant $ ilde{O}( sqrt{N_1/(N_0/mathtt{C}(pi^{-}| ho)+N_1)} )$ speed-up compared to pure online learning, where $mathtt{C}(pi^-| ho)$ is the concentrability coefficient over all sub-optimal policies. Our results also reveal an interesting separation on the desired coverage properties of the offline dataset for sub-optimality gap minimization and regret minimization. We further validate our theoretical findings in several experiments in special RL models such as linear contextual bandits and Markov decision processes (MDPs).

Problem

• Hybrid RL combining offline data and online interactions
• Optimal policy learning with improved performance metrics
• Analysis of coverage properties for different learning objectives

Innovation

• Hybrid RL combining offline and online data
• Confidence-based augmentation for optimal policy
• New concentrability coefficient for performance analysis