ROAD-VLA: Robust Online Adaptation via Self-Distillation for Vision-Language-Action Models

2026-06-24

AI Summary

This work addresses the challenge of online adaptation in vision-language-action (VLA) models under sparse reward settings, particularly the modality gap between symbolic instructions and low-level actions. To bridge this gap, the authors propose ROAD-VLA, an advantage-guided self-distillation framework that introduces calibrated advantage estimates at the action token logits level. This approach constructs a proximal teacher model aligned with the current policy, effectively transforming sparse environment rewards into dense, token-level supervision signals. The method further establishes a theoretical lower bound for policy improvement. Experimental results across seven robotic manipulation environments demonstrate that ROAD-VLA significantly outperforms PPO in nearly all settings, exhibiting superior online adaptation capabilities under both in-distribution and out-of-distribution shifts.

Abstract

Effective online adaptation of vision-language-action (VLA) models remains challenging, as sparse rewards provide weak supervision for high-dimensional autoregressive action policies. Although self-distillation can in principle provide denser training signals, we find that text-based privileged teachers conditioned on demonstrations, retrieved experiences, or high-level plans are ineffective for VLA adaptation, exposing a modality gap between symbolic guidance and low-level robot actions. We propose ROAD-VLA, an advantage-guided self-distillation framework that constructs a proximal teacher directly in action space by perturbing action-token logits with calibrated advantage estimates. This converts sparse rewards into dense token-level supervision while keeping the teacher close to the current policy. We further derive a policy-improvement lower bound under calibrated advantages and accurate teacher matching. Across seven robotic manipulation environments with in-distribution and out-of-distribution shifts, ROADVLA outperforms PPO in nearly all settings, demonstrating robust online VLA adaptation.

Problem

• vision-language-action models
• online adaptation
• sparse rewards
• modality gap
• robotic manipulation

Innovation

• self-distillation
• vision-language-action
• online adaptation
• advantage estimation
• robotic manipulation