Z-1: Efficient Reinforcement Learning for Vision-Language-Action Models

2026-06-30

AI Summary

This work addresses the limitation of existing vision-language-action (VLA) models, which predominantly rely on supervised fine-tuning and struggle to learn continuously from failures. The authors propose Z-1, the first reinforcement learning post-training framework tailored for streaming VLA models. Starting from an initial supervised fine-tuning phase using only publicly available RoboCasa demonstration data, Z-1 enables online performance improvement through task-level Grouped Relative Policy Optimization (GRPO). Key innovations include shared-prefix rollouts, tree-structured trajectory branching, completion-aware reward calibration, and selective joint training of vision-language and action modules, collectively enhancing training efficiency and stability. Evaluated across all 24 RoboCasa tasks, Z-1 achieves an average success rate of 80.6%, surpassing the baseline by 13.2 percentage points and outperforming the current public state-of-the-art.

Abstract

Vision-Language-Action (VLA) models offer a promising framework for robotic manipulation by connecting language instructions, visual observations, and continuous control. However, most existing policies remain limited by behavior cloning or supervised fine-tuning (SFT) from fixed demonstrations, which provides limited opportunity to improve from the policy's own failures. In this paper, we present Z-1, a reinforcement learning (RL) post-training framework for flow-based VLA models. Built on top of $π_{0.5}$, Z-1 uses only publicly released RoboCasa demonstrations for SFT and then applies a task-wise Group Relative Policy Optimization (GRPO) strategy across $24$ standard RoboCasa tasks. To improve the efficiency and stability of online optimization, Z-1 combines shared-prefix rollout construction, tree-structured trajectory branching, completion-aware reward calibration, and selective joint training of VLM and Action Expert. Across all $24$ RoboCasa tasks, Z-1 achieves an average success rate of $80.6\%$, improving over its SFT initialization by $13.2\%$ points and outperforms the published sota models. These results show that systematic GRPO post-training can substantially improve flow-based VLA policies without additional private demonstrations.

Problem

• Vision-Language-Action models
• reinforcement learning
• behavior cloning
• supervised fine-tuning
• robotic manipulation

Innovation

• Reinforcement Learning
• Vision-Language-Action Models
• Group Relative Policy Optimization
• Flow-based Policy
• Robotic Manipulation