WA-SpecDec: World-Aware Speculative Decoding for Vision-Language-Action Models

2026-08-09

AI Summary

This work addresses the high latency in existing vision–language–action models caused by autoregressive action generation and the limitations of conventional speculative decoding, which employs scene-agnostic fallback mechanisms prone to collisions or grasp failures in contact-sensitive regions. The paper introduces, for the first time, physical scene awareness into the speculative decoding framework: during the prefill stage, a world model constructs a shared world-perception state to generate action drafts and dynamically verify their feasibility. This approach enables context-aware tolerance adjustment without modifying the underlying lax acceptance rules, substantially improving both safety and efficiency. Experiments demonstrate consistent gains across three state-of-the-art lax acceptance schemes, achieving higher task success rates, 1.5× speedup at matched success rates, and an average 18.6% reduction in near-contact failure rates.

Abstract

Vision-language-action (VLA) policies generate robot controls autoregressively, making closed-loop latency dominated by repeated target-model forward passes. Speculative decoding reduces this cost by verifying blocks of draft action tokens in parallel, and recent VLA methods further relax token-level acceptance because small differences in action-token space often map to similar continuous controls. However, this relaxation remains scene-agnostic. A fixed token-distance tolerance treats the same action-token deviation as equally safe across states, although deviations that are harmless in free space can cause collisions or grasp failures near contact. We propose WA-SpecDec, a world-aware speculative decoding framework that injects world-model-derived physical scene awareness during the VLA prefill stage, producing shared world-aware prefill states for draft proposal and target verification without changing the relaxed acceptance rule. Across three state-of-the-art relaxed acceptance schemes, WA-SpecDec preserves higher task success under looser relaxation and enables longer accepted prefixes. At comparable-success operating points, WA-SpecDec achieves a 1.5x matched-success speedup over VLA speculative decoding alone and reduces near-contact failure (NCF) by 18.6% on average relative to the corresponding speculative baselines.

Problem

• speculative decoding
• vision-language-action models
• scene-awareness
• action-token tolerance
• robot control

Innovation

• world-aware speculative decoding
• vision-language-action models
• relaxed token acceptance
• physical scene awareness
• closed-loop robot control