Lost in Reconstruction: Aligning Action Representations with Language in Vision-Language-Action Models

2026-08-11

AI Summary

This work addresses the insufficient alignment between language and action in existing vision–language–action models, which typically optimize only raw action reconstruction loss and overlook the semantic execution cues embedded in action verbs. To remedy this, the authors propose SALT, a novel approach that leverages language instructions as supervisory signals within a VQ-VAE–style action tokenizer. Specifically, SALT introduces an auxiliary objective that employs a frozen vision–language model to reconstruct task instructions from discrete action latent variables. This design enables the action representations to retain high reconstruction fidelity while acquiring verb-specific semantic properties, thereby significantly improving language grounding. Evaluated on SimplerEnv, the SALT policy achieves a success rate of 71.9%, substantially outperforming the standard reconstruction-only VQ-VAE (42.7%) and FAST (31.2%).

Abstract

Action verbs describe not only the physical outcomes of actions, but also how those actions are performed. Yet action representations in vision-language-action models (VLAs) are typically optimized for reconstruction under L1/L2 losses in raw action space, where numerical proximity need not reflect linguistically meaningful distinctions. On BridgeV2, we show that action trajectories contain verb-grounding information beyond visual state changes, and that reconstruction-only discrete tokenization systematically erodes this information. To address this problem, we introduce SALT, a Semantically ALigned action Tokenizer that augments a VQ-VAE-style tokenizer with an auxiliary objective requiring a frozen vision-language model to recover the episode instruction from quantized action latents. Policies trained with SALT achieve 71.9% average success in SimplerEnv, compared with 42.7% for a reconstruction-only VQ-VAE tokenizer and 31.2% for FAST. SALT also develops verb-specialized codes while maintaining reconstruction fidelity. These results show that robot action trajectories provide a source of language grounding and that preserving this structure in action representations can substantially improve language-conditioned control.

Problem

• vision-language-action models
• action representation
• language grounding
• verb semantics
• action tokenization

Innovation

• vision-language-action models
• semantic alignment
• action tokenization
• language grounding
• VQ-VAE