DriveVLA-M0: Failure-Aware Memory Augmentation for Autonomous Driving

2026-08-10

AI Summary

This work addresses the challenge that existing end-to-end vision-language-action (VLA) autonomous driving models struggle to learn from past failures and tend to repeat errors under distribution shifts. The authors propose DriveVLA-M0, which introduces, for the first time, a failure-aware latent memory mechanism. This approach constructs a memory bank comprising failure cases, structured scene representations, and expert trajectories, and employs a retrieval model that disentangles road structure from dynamic interactions. During inference, retrieved knowledge is injected via lightweight LoRA-based test-time training (TTT) without modifying the backbone network, enabling targeted error correction. Evaluated on NAVSIMv1 and NAVSIMv2, DriveVLA-M0 achieves 94.1 PDMS and 47.0 EPDMS, respectively, with only 26.44 ms of TTT latency. Performance consistently improves with larger memory sizes, yielding online gains at zero training cost.

Abstract

Vision-Language-Action (VLA) models have recently emerged as a promising paradigm for end-to-end autonomous driving by enabling unified reasoning across perception, language, and planning. However, existing approaches lack mechanisms to exploit past failures or adapt to distribution shifts, causing the model to persistently underperform on similar scenarios where it has previously failed. In this paper, we propose DriveVLA-M0, a retrieval-augmented VLA with failure-aware latent memory. We construct a latent memory pool that stores failure cases along with their structure scene representations and expert trajectory labels, and design a dedicated Retrieve Model that decouples static road structure and dynamic agent interactions to enable structurally grounded retrieval. At inference time, retrieved cases are injected into the model via a lightweight decoupled LoRA-based test-time training (TTT) mechanism, allowing targeted and scenario-specific correction without modifying the backbone. Extensive experiments on NAVSIMv1 and NAVSIMv2 benchmark demonstrate that our approach consistently outperforms prior methods, achieving 94.1 PDMS on Navtest and 47.0 EPDMS on Navhard with only 26.44 ms TTT backward latency overhead. Furthermore, we show that DriveVLA-M0 scales effectively with additional memory, enabling training-free performance gains through memory expansion. The code is available at https://github.com/ZebinX/DriveVLA-M0.

Problem

• autonomous driving
• Vision-Language-Action
• failure awareness
• distribution shift
• memory augmentation

Innovation

• failure-aware memory
• retrieval-augmented VLA
• test-time training
• decoupled LoRA
• autonomous driving