Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving

2026-05-21

AI Summary

Existing end-to-end vision-language-action (VLA) models struggle to balance trajectory planning accuracy with inference efficiency: autoregressive approaches suffer from memory bandwidth constraints and exposure bias, while full-sequence diffusion models violate the causality between perception and planning and preclude KV cache reuse. This work proposes a block-diffusion VLA architecture that enables bidirectional optimization within semantic units while strictly preserving causal ordering across units. By freezing syntactic tokens of structured outputs as segment skeletons and integrating segment-aware training, skeleton-guided decoding, and a multi-trajectory fusion strategy with shared prefix KV caching, the model achieves both safety-critical planning fidelity and high throughput with low variance. It attains state-of-the-art ADE@3s/5s and the highest RFS on WOD-E2E, reduces L2 error to 0.32 meters on nuScenes (a 22% improvement), and delivers a 12× higher throughput than autoregressive baselines.

Abstract

End-to-end autonomous driving via Vision-Language-Action (VLA) models demands a precarious balance between high-fidelity trajectory planning and efficient inference. Existing paradigms typically fall short: autoregressive (AR) VLAs are memory-bandwidth-bound on edge hardware and prone to exposure-bias drift, while full-sequence diffusion models preclude KV-cache reuse and suffer from "logical leakage" that violates the fundamental perceive-then-plan causality. We present Fast-dDrive, a block-diffusion VLA that performs bidirectional refinement within semantic units while enforcing strict causal ordering across them. Leveraging the observation that driving VLAs often emit structured JSON-like outputs, Fast-dDrive freezes structural tokens into a section scaffold and employs a section-aware training recipe that prioritizes safety-critical planning. We further introduce Scaffold Speculative Decoding to achieve AR-equivalent quality at significantly higher throughput. Finally, we propose a low-overhead test-time scaling scheme: by forking $N$ stochastic trajectory rollouts from a single shared-prefix KV cache and averaging them, we effectively suppress prediction variance at a fractional computational cost. Empirical results demonstrate that Fast-dDrive redefines the speed-accuracy frontier for driving agents. On the WOD-E2E test set, Fast-dDrive achieves SOTA ADE@3s and ADE@5s, alongside the highest RFS among diffusion-based VLAs; on nuScenes, it reduces average L2 error to $0.32$m (a $22\%$ improvement). When integrated with SGLang, our framework delivers $12\times$ throughput speedup over the AR baseline, narrowing the gap between high-capacity VLAs and the efficiency demands of real-time on-vehicle deployment.

Problem

• autonomous driving
• Vision-Language-Action models
• diffusion models
• causality
• inference efficiency

Innovation

• block-diffusion
• causal ordering
• scaffold speculative decoding
• structured output modeling
• KV-cache reuse