Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning

2026-07-10

AI Summary

This work addresses the critical challenge of unreliable value estimation in offline-to-online reinforcement learning, which severely limits policy performance—particularly when leveraging heterogeneous robotic experiences. To this end, the authors propose Robo-ValueRL, a unified framework that systematically investigates the impact of value reliability on policy optimization for the first time. The approach introduces dual metrics—global progress and local preference—to assess value credibility and incorporates a history-conditioned value estimator, quality-conditioned consistency pretraining, a residual adaptation module, and a value-guided data prioritization mechanism, yielding an end-to-end analyzable training pipeline. Evaluated on 240 hours of offline data and over 3,000 online trajectories, the method achieves success rates of 86% and 84% on millimeter-precision chip insertion and general block disassembly tasks, respectively.

Abstract

Offline-to-online reinforcement learning is promising for generalizable robotic manipulation, yet its full-stack complexity obscures reproduction and diagnosis. Within such systems, value estimation plays a central role in prioritizing heterogeneous data for policy improvement. Despite its importance, the central question remains underexplored: how value-function reliability shapes policy optimization in offline-to-online reinforcement learning. To answer this question, we propose Robo-ValueRL, a unified framework that enables reliable value estimation and systematically traces its downstream effects on policy pretraining and online improvement. Concretely, Robo-ValueRL learns a history-conditioned value estimator and evaluates its reliability through global-progress and local-preference metrics. These resulting value estimates are propagated into quality-conditioned consistency-policy pretraining and a residual adaptation module on online rollouts, providing a unified testbed for analyzing how value reliability shapes downstream policy performance. Across 240 hours of offline demonstrations and over 3,000 online rollout trajectories, our extensive experiments show that downstream performance is strongly associated with value reliability. Reliable value functions provide better action-quality estimates, allowing value-guided offline RL to scale more effectively than quality-agnostic behavior cloning, and stabilize online improvement by prioritizing high-quality rollout data. Integrating reliable value guidance through offline pretraining with online improvement, our system achieves 86% success on millimeter-level precise chip insertion and 84% on generalizable block disassembly. We hope these findings highlight the importance of value-guided data utilization for effective policy improvement from heterogeneous robotic experience.

Problem

• offline-to-online reinforcement learning
• value estimation
• policy optimization
• robotic manipulation
• value reliability

Innovation

• reliable value estimation
• offline-to-online reinforcement learning
• history-conditioned value estimator
• quality-conditioned policy pretraining
• value-guided data utilization