NAC: Neural Action Codec for Vision-Language-Action Models

2026-06-19

AI Summary

This work addresses the challenge that existing discrete action tokenizers in vision–language–action models struggle to simultaneously achieve high compression ratios, low latency, and strong downstream task performance. Inspired by neural audio codecs, the authors propose a multi-scale residual vector-quantized generative adversarial network (RVQGAN) that treats robot action trajectories as multi-channel one-dimensional signals for efficient compression and reconstruction. Key innovations include a shifted codebook architecture that yields a structured and compact action token space, a Vocos-style decoder with an inverse short-time Fourier transform (ISTFT) head, and tailored time-domain and non-Mel spectral reconstruction losses combined with an adversarial discriminator. Evaluated on LIBERO-10, RoboMimic, and real-world tasks, the method achieves significantly lower reconstruction error and higher task success rates than conventional binning, FAST, and existing vector-quantization tokenizers, at comparable or better compression ratios.

Abstract

Vision-language-action (VLA) models rely on discrete action tokenizers to bridge continuous robot control and autoregressive sequence modeling, yet existing tokenizers often trade off between compression, latency, and downstream performance. We revisit this design through the lens of neural audio codecs-convolutional encoder-decoder architectures with residual vector quantization that serve as the standard front end for audio foundation models. Motivated by their success, we introduce the Neural Action Codec (NAC), which treats short robot action trajectories as multi-channel 1D signals and compresses them using a multi-scale RVQGAN architecture. We observe that audio-specific mel-spectrogram objectives are ill-suited for kinematic signals; however, by replacing them with simple time-domain and non-mel spectral reconstruction losses, audio-codec-style models can autoencode actions with high fidelity without substantial architectural changes. NAC provides a compact, ordered token space via offset codebooks, enabling standard autoregressive policies to operate over short, structured sequences. Meanwhile, a Vocos-style decoder with an ISTFT head and adversarial discriminators recovers smooth, detailed trajectories. Across LIBERO-10, RoboMimic, and a suite of real-world manipulation tasks, NAC achieves lower reconstruction error and higher success rates than binning, FAST, and prior VQ-based tokenizers at comparable or better compression rates. These results demonstrate that repurposed neural audio codecs offer a strong, practical backbone for learned action tokenization in modern VLAs.

Problem

• action tokenization
• vision-language-action models
• robot control
• compression-latency trade-off
• discrete action representations

Innovation

• Neural Action Codec
• Residual Vector Quantization
• Action Tokenization
• Vision-Language-Action Models
• RVQGAN