Teaching Vision-Language-Action Models What to See and Where to Look

2026-07-01

AI Summary

This work addresses the overreliance on textual reasoning in existing vision-language-action models for autonomous driving, which often neglects critical spatial dependencies and leads to unreliable trajectory predictions. To remedy this, the authors propose a novel vision-guided learning framework that explicitly models the decision chain of “what to see—where to look—how to act.” The approach integrates Driving-aware Vision Distillation (DVD) to inject driving-specific perceptual priors and introduces 2D Trajectory-Guided Prompts (2D-TGP) to provide spatial guidance aligned with feasible trajectories. By combining supervised fine-tuning with GRPO reinforcement learning, the method uniquely fuses driving priors and trajectory-aware visual prompts, significantly enhancing alignment between actions and visual observations. Evaluated on the NAVSIM and nuScenes benchmarks, the proposed framework achieves state-of-the-art performance, markedly improving both accuracy and robustness in trajectory prediction.

Abstract

Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.

Problem

• Vision-Language-Action models
• autonomous driving
• spatial dependencies
• trajectory prediction
• visual representation

Innovation

• Vision-Language-Action Models
• Driving-aware Vision Distillation
• Trajectory-Guided Prompts
• Spatial Conditioning
• Autonomous Driving