RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance

2026-08-10

AI Summary

This work addresses the challenge of generalizing reward models across diverse robotic platforms using large-scale, heterogeneous data, a task hindered by existing methods’ reliance on in-task preference or progress annotations. The paper introduces, for the first time, temporal distance—the directed cost from a current observation to a language-specified goal—as a scalable, cross-platform value supervision signal that requires no human annotation and can be learned from 7,000 hours of video data. To prevent shortcut learning, the approach employs random temporal sampling, temporal shuffling, and a value-isolating attention mechanism. Evaluated on RBM-EVAL-OOD, the model achieves a Kendall’s τ_a of 0.675, surpassing state-of-the-art fully preference-supervised methods. It also improves online policy success rates from 52.5% to 72.5% and offline success rates from 63.8% to 82.5%.

Abstract

General-purpose reward models are increasingly the bottleneck for scaling robot learning, yet the recipe for learning value-related capabilities from large-scale heterogeneous corpora remains underexplored. Existing approaches tie supervision to task-internal anchors such as preferences or normalized progress, none of which transfer cleanly across embodiments and data sources. We introduce RynnValue, an open-source value foundation model for robotic manipulation that replaces these anchors with temporal distance, the directed cost-to-go from an observation to the language-specified goal. Because temporal-distance labels can be derived directly from timestamps, RynnValue scales to over 7,000 hours and roughly 3M instruction-conditioned clips without preference or progress annotations. To make temporal-value learning reliable at scale, we combine random temporal sampling, temporal-order shuffling, and value-isolation attention, suppressing shortcuts that would leave predictions insensitive to failures and regressions. Trained without preference labels, RynnValue attains an average Kendall's tau_a of 0.675 on RBM-EVAL-OOD, surpassing the fully preference-supervised state of the art (0.655) and more than doubling a progress-only counterpart (0.292), while generalizing zero-shot to unseen tasks, embodiments, and viewpoints. Converted into dense rewards via potential-based shaping, it raises real-world policy success from 52.5% to 72.5% online and from 63.8% to 82.5% offline. These results establish temporal distance as a scalable supervision target and practical reward interface for generalist robot policies.

Problem

• reward models
• robotic learning
• value foundation models
• temporal distance
• heterogeneous corpora

Innovation

• temporal distance
• value foundation model
• robotic manipulation
• preference-free learning
• reward shaping