LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving

2026-06-29

AI Summary

Existing vision-language models generate driving trajectories that convey only coarse intent, lacking geometric precision, future scene awareness, and multi-view consistency. To address these limitations, this work proposes LWDrive, a hierarchical framework that first interprets vision-language model outputs as high-level driving intents and then refines trajectories through a coarse-to-fine optimization process within a diverse candidate space using a hierarchical world model. The approach innovatively incorporates supervision from generated future frames to encourage the learning of forward-looking scene representations. Furthermore, it introduces a cascaded hierarchical planner that fuses multi-scale features with multi-view bird’s-eye-view (BEV) representations, leveraging action query embeddings and a scoring-based selection mechanism to refine trajectories. The method achieves state-of-the-art performance on the NAVSIM and NAVSIM-v2 benchmarks, attaining scores of 92.0 and 89.6, respectively.

Abstract

Vision-Language Models (VLMs) provide powerful semantic understanding and commonsense reasoning for End-to-End Autonomous Driving (E2E-AD) planning. However, trajectories directly generated by VLMs often encode only coarse driving intentions and remain insufficient for geometrically accurate, future-aware, and multi-view-grounded planning. To address these limitations, we develop the Layer-Wise World-Model-Guided Driving framework (LWDrive). LWDrive is a VLM planning framework that refines coarse trajectories through layer-wise world-model guidance. Instead of treating the VLM output as the final trajectory, LWDrive uses it as an intent-aware coarse plan, expands a diverse candidate space around it, and progressively refines the candidates through a Foresight Cascade Planner (FCP). Specifically, we introduce future-frame generation supervision to encourage the VLM to learn forward-looking scene representations, thereby injecting planning-relevant predictive dynamics into its internal hidden states. Built upon these world-model-supervised representations, FCP exploits VLM features across multiple layers and integrates historical temporal states, Action-Query representations, and current-frame multi-view Bird's-Eye-View (BEV) features to refine candidate trajectories in a coarse-to-fine manner. This design enables progressive correction of spatial positions and motion trends while grounding trajectory refinement with multi-view scene cues and preserving the high-level driving intention produced by the large model. Finally, a score head evaluates the refined candidates and selects the best trajectory as the final planning output. Experiments show that LWDrive achieves a score of 92.0 on the NAVSIM benchmark and 89.6 on NAVSIM-v2. Code and models will be made publicly available.

Problem

• Vision-Language Models
• Autonomous Driving
• Trajectory Planning
• World Model
• Multi-view Perception

Innovation

• Layer-wise refinement
• World-model guidance
• Vision-Language Model (VLM)
• Foresight Cascade Planner
• Multi-view BEV