Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes

2026-06-15

AI Summary

This work addresses the challenge of online reinforcement learning fine-tuning for pretrained vision-language-action models under sparse binary rewards, where existing approaches struggle to provide effective per-timestep supervision and conflate feasibility with efficiency objectives, leading to credit assignment errors in mixed intervention trajectories. The authors propose Hierarchical Advantage-Weighted Behavioral Cloning (HABC), which trains separate critic heads for feasibility and efficiency on distinct data subsets and employs a state-adaptive gating mechanism to dynamically fuse their advantages into fine-grained actor loss weights. Additionally, an intervention-aware credit assignment scheme assigns outcome labels only to autonomously executed segments. By decoupling and dynamically integrating these two advantage signals for the first time, HABC substantially improves success rates on three real-world, contact-rich bimanual tasks—from 36%, 44%, and 12% to 92%, 88%, and 38%, respectively.

Abstract

When pretrained VLA policies are fine-tuned through online RL, each rollout episode produces only a single binary outcome (success or failure), yet the actor update requires per-transition supervision. Existing approaches commonly reduce this sparse outcome to a single scalar reward or advantage signal, which conflates distinct forms of transition-level feedback and provides limited guidance once basic task success becomes achievable. First, a single scalar signal conflates the two objectives of viability and efficiency; once basic success is achieved, the binary label provides no gradient to distinguish efficient completions from slow ones. Second, real-world rollouts mix autonomous and intervention segments; naively assigning episode outcomes across these boundaries introduces incorrect credit assignment. To address these issues, we propose Hierarchical Advantage-Weighted Behavior Cloning (HABC), which trains separate critic heads for these two objectives on different data subsets and combines their outputs with a state-adaptive balance. A state-adaptive gate $g_t$ merges their one-step advantages, prioritizing viability when success is uncertain and shifting to efficiency only when viability is high, and converts the result into per-transition weights on the actor loss. Intervention-aware credit assignment further restricts outcome labels to segments executed by the current policy, preventing supervision from leaking across intervention boundaries. In real-robot experiments on three contact-rich bimanual tasks, HABC raises success from supervised fine-tuning (SFT) baselines of 36%, 44%, and 12% to 92%, 88%, and 38%.

Problem

• sparse rewards
• credit assignment
• online RL fine-tuning
• VLA policies
• hierarchical advantage

Innovation

• Hierarchical Advantage Weighting
• Online RL Fine-Tuning
• Sparse Reward
• Intervention-Aware Credit Assignment
• VLA Policies