OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models

2026-04-20

AI Summary

This work addresses the challenge of end-to-end autonomous driving, which requires joint optimization of heterogeneous tasks such as language generation, object detection, and trajectory prediction. Existing approaches suffer from architectural fragmentation and limited backbone reuse due to separate or cascaded decoders. To overcome this, the authors propose a unified causal Transformer decoder built upon a pretrained vision-language model, which embeds images, textual inputs, and structured trajectory queries into a shared sequence. This enables multitask co-training through a common attention mechanism, unifying perception, planning, and language generation within a single decoder for the first time. The method significantly enhances cross-modal transfer efficiency. Experiments show state-of-the-art performance with a 0.28 L2 error and 0.18 collision rate on nuScenes open-loop evaluation, a PDMS score of 86.8 in NAVSIM closed-loop simulation, and approximately 40% lower inference latency.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Vision-Language Models(VLMs) excel at autoregressive text generation, yet end-to-end autonomous driving requires multi-task learning with structured outputs and heterogeneous decoding behaviors, such as autoregressive language generation, parallel object detection and trajectory regression. To accommodate these differences, existing systems typically introduce separate or cascaded decoders, resulting in architectural fragmentation and limited backbone reuse. In this work, we present a unified autonomous driving framework built upon a pretrained VLM, where heterogeneous decoding behaviors are reconciled within a single transformer decoder. We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling. By organizing visual and structured query tokens within a single causal decoder, structured queries can naturally condition on visual context through the original attention mechanism. Textual and structured outputs share a common attention backbone, enabling stable joint optimization across heterogeneous tasks. Trajectory planning is realized within the same causal LLM decoder by introducing structured trajectory queries. This unified formulation enables planning to share the pretrained attention backbone with images and perception tokens. Extensive experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance, including 0.28 L2 and 0.18 collision rate on nuScenes open-loop evaluation and competitive results (86.8 PDMS) on NAVSIM closed-loop evaluation. The full model preserves multi-modal generation capability, while an efficient inference mode achieves approximately 40% lower latency. Code and models are available at https://github.com/Z1zyw/OneDrive

Problem

• autonomous driving
• multi-task learning
• heterogeneous decoding
• architectural fragmentation
• structured outputs

Innovation

• Vision-Language-Action Models
• Unified Transformer Decoder
• Structured Query Tokens
• End-to-End Autonomous Driving
• Pretrained Attention Transfer