Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance

2024-10-17

arXiv.org

AI Summary

To address the performance degradation of general-purpose robotic policies when deployed under heterogeneous data quality and uncertain cross-platform applicability, this paper proposes Value-Guided Policy Steering (V-GPS). V-GPS leverages only the value function trained via offline reinforcement learning to re-rank actions generated by a pre-trained general policy during inference—requiring no fine-tuning, no access to policy weights, and no knowledge of its internal architecture. To our knowledge, V-GPS is the first method enabling plug-and-play performance enhancement across diverse policy architectures, training datasets, and robotic platforms. We evaluate V-GPS on five state-of-the-art general-purpose policies, 12 benchmark tasks, and multiple physical and simulated robot platforms. Results demonstrate consistent and significant improvements in both action accuracy and task success rate. V-GPS thus establishes a novel zero-cost deployment optimization paradigm for general-purpose robotic policies.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Large, general-purpose robotic policies trained on diverse demonstration datasets have been shown to be remarkably effective both for controlling a variety of robots in a range of different scenes, and for acquiring broad repertoires of manipulation skills. However, the data that such policies are trained on is generally of mixed quality -- not only are human-collected demonstrations unlikely to perform the task perfectly, but the larger the dataset is, the harder it is to curate only the highest quality examples. It also remains unclear how optimal data from one embodiment is for training on another embodiment. In this paper, we present a general and broadly applicable approach that enhances the performance of such generalist robot policies at deployment time by re-ranking their actions according to a value function learned via offline RL. This approach, which we call Value-Guided Policy Steering (V-GPS), is compatible with a wide range of different generalist policies, without needing to fine-tune or even access the weights of the policy. We show that the same value function can improve the performance of five different state-of-the-art policies with different architectures, even though they were trained on distinct datasets, attaining consistent performance improvement on multiple robotic platforms across a total of 12 tasks. Code and videos can be found at: https://nakamotoo.github.io/V-GPS

Problem

• Improving robotic foundation models
• Enhancing generalist robot policies
• Value-guided policy steering application

Innovation

• Value-Guided Policy Steering
• Offline RL value function
• Action re-ranking technique