PixelPilot: Scalable Vision-Language-Action Models for End-to-End Autonomous Driving

2026-07-05

AI Summary

This work addresses the limitations of existing vision-language action models in end-to-end autonomous driving, which rely on 2D-to-3D trajectory prediction and suffer from camera-parameter coupling, poor cross-dataset generalization, and degeneration into trivial solutions based solely on ego-vehicle states. To overcome these issues, the authors propose PixelPilot, a novel decoupled 2D-to-2D planning framework followed by deterministic 3D lifting: trajectories are first planned in the image plane in a sensor-agnostic manner and then lifted to 3D space during inference. The approach integrates knowledge injection and a dense intermediate reward mechanism based on Group Relative Policy Optimization (GRPO) to strengthen the causal link between visual perception and action decision-making. PixelPilot achieves state-of-the-art performance in both open-loop and closed-loop evaluations, significantly enhancing cross-dataset generalization and visual reasoning capabilities.

Abstract

Vision-Language-Action Models (VLAs), which leverage the advanced reasoning capabilities of Vision-Language Models (VLMs), show promising generalization in complex autonomous driving scenarios. Existing VLAs typically predict and optimize 3D trajectories from 2D images. While intuitive, this 2D-to-3D prediction is inherently entangled with camera parameters, leading to limited data scalability across heterogeneous driving datasets. Moreover, directly optimizing in 3D space induces severe convergence to trivial solutions, where VLAs rely on ego-status rather than visual scene understanding. To address these issues, we propose PixelPilot, a novel VLA featuring a decoupled planning and lifting paradigm. In the planning phase, PixelPilot reformulates scene understanding and trajectory prediction as sensor-agnostic 2D-to-2D tasks in the image plane, thereby facilitating scalable training across diverse datasets. The planned 2D trajectories are then deterministically lifted to 3D only during inference, ensuring the full exploitation of visual cues and generalization across different vehicles. To realize this paradigm, we propose a knowledge-instilled policy learning strategy that applies dense, intermediate rewards via Group Relative Policy Optimization (GRPO) to enforce a rigorous causal chain from visual perception to spatial planning. Extensive experiments demonstrate that PixelPilot achieves state-of-the-art performance in both open-loop and closed-loop settings, validating its superior scalability and visual reasoning capabilities.

Problem

• Vision-Language-Action Models
• autonomous driving
• 2D-to-3D prediction
• data scalability
• visual scene understanding

Innovation

• Vision-Language-Action Models
• Decoupled Planning and Lifting
• 2D-to-2D Trajectory Prediction
• Group Relative Policy Optimization
• Scalable Autonomous Driving