Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors

2026-06-24

AI Summary

This work addresses the limitations of existing reinforcement learning methods in effectively leveraging offline priors—such as offline datasets, pretrained policies, or multimodal foundation models—due to mismatches between deployment environments and the dynamic validity of such priors, which often constrain the performance of general-purpose policies. To overcome this, the paper introduces a diagnostic-driven online reinforcement learning framework that continuously evaluates the current effectiveness of priors and adaptively modulates their integration with online learning. The framework systematically identifies three functional roles priors play during optimization and reveals a previously uncharacterized “help-to-harm” inversion phenomenon, thereby moving beyond conventional one-size-fits-all integration paradigms. Extensive experiments—including control studies, cross-domain analyses, and adversarial ablations—demonstrate that the proposed approach significantly enhances learning efficiency and robustness in real-world deployment, establishing a new adaptive integration principle that surpasses standard benchmarking practices.

Abstract

Online reinforcement learning (RL) agents increasingly depend on knowledge acquired offline to achieve practical efficiency. Originally studied in offline-to-online RL, this paradigm now spans foundation model post-training and embodied intelligence, with prior types expanding from offline datasets and pre-trained policies to increasingly diverse knowledge sources such as multimodal foundation models and generative world models. Offline priors have become central to how deep RL is developed and deployed. However, this reliance introduces a challenge that the prevailing benchmark-driven paradigm cannot resolve: because prior validity varies across deployments and shifts during training, no single approach to managing it is universally optimal, and benchmark rankings offer limited guidance for real-world deployments. Rather than pursuing universal solutions, we argue that the field should shift to diagnosis-driven tension management, in which deployment-specific evidence guides how the learner relates to its priors throughout training, enabling both flexible and adaptive deployment. We support this position with a framework characterizing how priors reshape online optimization through three functional roles, controlled experiments demonstrating help-or-hurt reversals, cross-domain evidence from foundation model post-training to embodied intelligence, and engagement with five substantive counterarguments.

Problem

• offline priors
• online reinforcement learning
• deployment-specific adaptation
• prior validity
• diagnosis-driven

Innovation

• diagnosis-driven RL
• offline priors
• online reinforcement learning
• adaptive deployment
• foundation models