Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning

2026-08-11

AI Summary

This work addresses a critical limitation in offline-to-online reinforcement learning: offline pre-trained critics often introduce biased value estimates during online fine-tuning, which hinders policy optimization and exploration efficiency. To mitigate this issue, the paper proposes a novel paradigm that entirely discards critic pre-training in the offline phase and instead initializes the critic from scratch during online fine-tuning. This approach effectively prevents the propagation of offline-induced bias and enables faster adaptation to the true environment dynamics. The method is compatible with mainstream offline-to-online (O2O) frameworks and consistently matches or surpasses existing approaches across a range of complex tasks, achieving notably superior fine-tuning performance—particularly in challenging environments.

Abstract

Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce \textbf{C}ritic-\textbf{F}ree \textbf{P}retraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.

Problem

• offline-to-online reinforcement learning
• critic misalignment
• policy fine-tuning
• value estimation bias
• online adaptation

Innovation

• Critic-Free Pretraining
• Offline-to-Online Reinforcement Learning
• Online Fine-Tuning
• Value Estimation Bias
• Efficient Exploration