Learning More from Less: Reinforcement Learning from Hindsight

2026-07-09

AI Summary

This work addresses the challenge of low sample efficiency in reinforcement learning (RL) fine-tuning of vision-language-action (VLA) models, particularly in sparse-reward manipulation tasks where frequent failures hinder effective learning. The study introduces hindsight relabeling into VLA RL fine-tuning for the first time, leveraging vision-language models to semantically reinterpret failed trajectories and automatically generate new instructions and rewards. By jointly optimizing both original and relabeled trajectories, the method substantially improves sample utilization. It achieves a fivefold gain in sample efficiency on out-of-distribution tasks from the LIBERO-PRO benchmark, outperforming dense progress-based reward baselines. The approach demonstrates consistent effectiveness across diverse VLA backbone architectures and validates successfully on a real-world Franka robotic platform.

Abstract

Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, but every update consumes robot rollouts that are slow and costly to collect, making sample efficiency a central concern. Manipulation tasks typically provide only sparse rewards, so a weak policy fails almost every rollout early in training and has little to learn from, even when those failures execute coherent behavior. Such a failure, however, is a success at a different task. We present Learning from Hindsight (LfH), which brings hindsight relabeling to RL post-training of VLAs by scoring failed rollouts against the tasks they actually achieved. A single vision-language model relabels both the instruction and the reward, proposing a hindsight instruction for a group of failed rollouts and scoring how well each satisfies it, and the policy trains on the relabeled and original rollouts jointly. Because VLAs generalize across language, relabeling in language lets the policy learn more from the same trajectories. On out-of-distribution LIBERO-PRO tasks, where standard RL improves only slowly, LfH achieves $5\times$ improvement in sample efficiency, and outperforms a dense progress-reward baseline. The gains hold across VLA backbones and on a physical Franka robot.

Problem

• reinforcement learning
• sample efficiency
• sparse rewards
• vision-language-action models
• hindsight relabeling

Innovation

• hindsight relabeling
• vision-language-action models
• sample efficiency
• reinforcement learning
• language-conditioned robotics