SA-VLA: State-aware tokenizer for improving Vision-Language-Action Models' performance

2026-06-29

AI Summary

This work addresses the challenge that discrete action tokenization in vision–language–action (VLA) models struggles to accurately reconstruct continuous control signals conditioned on robot proprioceptive states, such as joint configurations and object poses. To overcome this limitation, the authors propose SA-VLA, a state-aware action tokenizer that integrates current state information into the discrete action decoding process. This is achieved through either state–action cross-attention or a lightweight state adapter, enabling each discrete token to represent a family of state-dependent continuous actions. By conditioning action generation on real-time robot states, SA-VLA significantly enhances representational capacity while preserving the efficiency of discrete modeling. Empirical results demonstrate substantial improvements: average success rates on 12 RoboTwin tasks rise from 0.29 to 0.56, and zero-shot sim-to-real transfer performance on three real-world tasks improves from 0.15 to 0.33.

Abstract

Discrete action tokenization provides a compact interface for autoregressive VLA policies, but accurately recovering continuous robot actions from discrete codes remains challenging. Existing tokenizers typically map each discrete code to a fixed continuous action prototype, ignoring the robot's current proprioceptive state. This limitation is particularly pronounced in manipulation, where the same action token may require different continuous controls under different joint configurations, object poses, and contact conditions. We therefore propose SA-VLA, a state-aware action tokenizer that conditions action decoding on robot state. We study two state-injection mechanisms for VQ-based action tokenization: cross-attention between state and action features, and a lightweight state adapter that predicts action-wise modulation factors for state-conditioned action modulation and reconstruction. The adapter formulation expands the effective support of a finite codebook by allowing each discrete token to represent a family of state-dependent continuous actions, while preserving the efficiency and compatibility of discrete action modeling. Integrated into an LLM-based VLA policy, SA-VLA supports both autoregressive and parallel action-token decoding with minimal changes to the model interface. On 12 RoboTwin manipulation tasks, SA-VLA improves the average success rate from 0.29 to 0.56 over the strongest tokenizer baseline. In zero-shot sim-to-real experiments on three real-world tasks, it further improves average success from 0.15 to 0.33 over the strongest tokenizer baseline. These results demonstrate that state-conditioned action decoding is a simple and effective mechanism for reducing the compression gap in discrete VLA policies.

Problem

• Vision-Language-Action Models
• discrete action tokenization
• state-aware decoding
• robot manipulation
• action reconstruction

Innovation

• state-aware tokenization
• vision-language-action models
• discrete action representation
• robot manipulation
• sim-to-real transfer