X-Tokenizer: A Multimodal Action Tokenizer for Vision-Language-Action Pretraining

2026-06-07

AI Summary

This work addresses the limitation of existing action tokenizers, which prioritize action reconstruction yet lack semantic supervision bridging vision–language reasoning and continuous robotic control. To overcome this, we propose X-Tokenizer, a lightweight encoder–semantic residual quantization (SRQ)–decoder architecture that models action tokenization as a semantic interface between multimodal reasoning and executable control, enabling a unified action representation across platforms. The key innovation lies in the SRQ mechanism: the initial layer learns coarse-grained intent through masked action modeling, while deeper layers preserve fine-grained motion details. The model is pretrained via contrastive alignment and next-frame prediction. Evaluated on both real-world tasks and RoboTwin 2.0 simulation, X-Tokenizer outperforms FAST by 13.5% on multimodal grounding tasks and achieves an 8.25% improvement in long-horizon task performance.

Abstract

Modern Vision-Language-Action (VLA) models must bridge pretrained vision-language reasoning and precise continuous robot control. Existing action tokenizers discretize actions primarily for reconstruction, producing codes that preserve motion geometry but provide only weak semantic supervision to the backbone. We therefore formulate action tokenization not as mere compression, but as semantic interface learning between multimodal reasoning and executable control. To this end, we introduce X-Tokenizer, a lightweight encoder-Semantic Residual Quantization (SRQ)-decoder architecture that provides a shared action interface across diverse robotic arm embodiments. Its key component, SRQ, imposes an asymmetric structure on residual vector quantization: the first level is trained with Masked Action Modeling (MAM) to form a discrete action language that captures coarse motion intent, while deeper levels remain reconstruction-oriented residuals that preserve fine-grained details. To further align action tokens with multimodal semantics, X-Tokenizer is pretrained with contrastive alignment to the representation space of a pretrained foundation model and with next-frame vision-language feature prediction. Pretrained on 2.4M trajectories (2.0B action frames), a single frozen X-Tokenizer plugs into a mixed discrete-continuous VLA as a representation-shaping supervision signal. X-Tokenizer achieves top real-world aggregate and strong RoboTwin 2.0 simulation results. Outperforming FAST in multimodal grounding (+13.5%) and long-horizon tasks (+8.25), it shows that action tokenizers serve as semantic interfaces for VLA pretraining beyond mere action compression.

Problem

• action tokenization
• vision-language-action
• semantic interface
• robotic control
• multimodal reasoning

Innovation

• X-Tokenizer
• Semantic Residual Quantization
• Vision-Language-Action Pretraining
• Masked Action Modeling
• Multimodal Semantic Alignment