FlashDrive: Flash Vision-Language-Action Inference for Autonomous Driving

2026-08-13

AI Summary

This work addresses the high inference latency of existing Vision-Language-Action models, which stems from their four-stage cascaded computation and hinders real-time deployment in autonomous driving. To overcome this limitation, the authors propose a co-optimized algorithm-system framework that introduces lightweight algorithmic shortcuts across all stages: cross-frame KV cache reuse for visual encoding, non-autoregressive diffusion drafters with speculative decoding for language prefilling, velocity-field-adaptive step-size caching for inference generation, and stream-matching denoising. These are synergistically combined with system-level optimizations including CUDA Graph compilation, operator fusion, and W4A8 quantization. Evaluated on a single GPU, the approach reduces end-to-end latency of the Alpamayo 1.5-10B model from 717 ms to 151 ms—a 4.7× speedup—enabling a 6.6 Hz inference rate while preserving trajectory accuracy and significantly lowering collision and off-road rates in simulation.

Abstract

Vision-Language-Action (VLA) models promise to bring end-to-end reasoning to autonomous driving, but their computational cost remains far too high for real-time control. The core challenge is structural: VLA inference is not a single bottleneck but a cascade of four. Visual encoding wastes compute on overlapping video frames; language-model prefill recomputes context that could be carried over from the previous timestep; reasoning tokens are generated serially despite low entropy; and flow-matching denoising applies uniform compute to a non-uniform velocity field. Addressing any one stage in isolation leaves the others untouched. We propose FlashDrive, an algorithm-system co-design framework that targets all four stages simultaneously. Our key insight is that each bottleneck admits a distinct, lightweight algorithmic shortcut: temporal overlap enables streaming KV-cache reuse across frames; the low per-token entropy and strong intra-block correlations of driving-domain reasoning make a non-autoregressive diffusion drafter highly effective for speculative decoding; and the velocity field's structure---sharp at the endpoints, flat in the middle---permits adaptive step caching that concentrates compute where it matters. Layered on system-level CUDA Graph compilation and kernel fusion, these techniques compound. Applied to Alpamayo 1.5-10B with W4A8 quantization, FlashDrive reduces end-to-end latency from 717ms to 151ms (4.7x) while leaving accuracy essentially unchanged: minADE6@6.4s shifts by only 0.08m, minADE1 improves, and closed-loop collision and off-road rates improve in simulation. By raising a 10B-parameter reasoning VLA from 1.4~Hz to 6.6~Hz on a single GPU, FlashDrive moves end-to-end autonomous driving substantially closer to real-time deployment.

Problem

• Vision-Language-Action
• autonomous driving
• real-time inference
• computational bottleneck
• end-to-end reasoning

Innovation

• Vision-Language-Action
• speculative decoding
• adaptive step caching
• KV-cache reuse
• algorithm-system co-design