Active Offline-to-Online Reinforcement Learning

2026-07-13

AI Summary

This work addresses a key challenge in offline-to-online reinforcement learning: how to efficiently select and fine-tune policies under a limited online interaction budget while avoiding performance degradation due to algorithmic or hyperparameter sensitivity. The paper introduces the first active policy selection framework tailored for such constrained settings. By constructing an upper confidence bound based on a local linear performance prediction model, the method dynamically balances resource allocation between online evaluation and fine-tuning, adaptively identifying the most promising policies for optimization. This approach overcomes the limitations of conventional strategies that either deploy a single policy or uniformly distribute the budget across candidates. Empirical results across multiple environments demonstrate that the proposed method significantly outperforms existing baselines, achieving more efficient utilization of scarce online interaction resources.

Abstract

Background: Offline reinforcement learning (RL) enables effective policies to be trained from large, previously collected datasets and subsequently improved through limited online interaction. This offline-to-online RL (O2O-RL) paradigm is particularly promising in nonstationary domains where interaction is costly or potentially hazardous. Standard O2O-RL pipelines train multiple candidate policies offline, evaluate them using off-policy or online evaluation, and then deploy and fine-tune the policy with the highest estimated value. However, as in offline pretraining, fine-tuning performance is highly sensitive to the choice of algorithm and hyperparameters, making it risky to commit to a single policy. Objectives: We study active policy selection for fine-tuning under a limited interaction budget in O2O-RL settings. To our knowledge, this is the first work to address this problem. Methods: We formulate the problem by identifying a fundamental trade-off between allocating online interactions to policy evaluation, which helps identify high-performing policies, and allocating them to fine-tuning, which improves policy performance. We then propose an approach that balances this trade-off by actively selecting policies for fine-tuning based on upper-confidence bounds on their future performance. These bounds are derived from locally linear performance forecasts fitted to observations obtained through online evaluation. Results: Across a diverse range of experiments, the proposed approach consistently outperforms existing O2O-RL baselines. Conclusions: Actively selecting and fine-tuning policies uses limited online interaction budgets more effectively than either committing to a single policy or dividing the budget equally among all policies. Our framework also advances offline RL toward practical deployment in real-world systems where online interaction is costly or risky.

Problem

• offline-to-online reinforcement learning
• policy selection
• limited interaction budget
• fine-tuning
• nonstationary domains

Innovation

• Active Policy Selection
• Offline-to-Online Reinforcement Learning
• Upper-Confidence Bound
• Limited Interaction Budget
• Performance Forecasting