Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning

2026-08-13

AI Summary

This work addresses the issue of erroneous credit assignment in vision-language-action reinforcement learning, where effective early-stage actions are unfairly penalized due to trajectory-level evaluation. To resolve this, the authors propose a stage-aware advantage estimation mechanism that detects task phases, aligns trajectories to phase-specific action intervals, and performs comparisons only among trajectories within the same phase, enabling fine-grained credit assignment. Built upon the GRPO framework, the method integrates task phase detection, phase-aligned trajectory processing, and phase-specific advantage computation to effectively mitigate credit aliasing. Experiments demonstrate significant improvements in task success rate and sample efficiency on RoboTwin 2.0, precise performance gains in divergent phases on LIBERO-Long, and preserved capability in shared initial phases.

Abstract

Outcome-driven reinforcement learning offers a scalable way to post-train vision-language-action (VLA) policies from sparse task-success feedback. In common GRPO-based VLA post-training, one rollout-level advantage is applied to every action in the trajectory. A rollout that completes several valid stages but fails later can therefore penalize the actions that produced its earlier progress. We call this trajectory-level credit aliasing. Temporal GRPO addresses this problem by constructing detectable task stages, aligning each rollout with stage-specific action intervals, and comparing only rollouts that have entered the same stage. The resulting stage advantages are applied to their corresponding intervals in a single policy update. On RoboTwin 2.0, Temporal GRPO improves task success and sample efficiency, with consistent gains across task horizons. Controlled updates on LIBERO-Long preserve shared prerequisite stages and concentrate improvement at the first stage where rollout outcomes diverge.

Problem

• trajectory-level credit aliasing
• vision-language-action
• reinforcement learning
• credit assignment
• sparse reward

Innovation

• Temporal GRPO
• credit assignment
• vision-language-action
• stage-aware reinforcement learning
• trajectory-level credit aliasing