PRIX: Learning to Plan from Raw Pixels for End-to-End Autonomous Driving

2025-07-23

AI Summary

End-to-end autonomous driving models often face deployment challenges on mass-market vehicles due to large model size, LiDAR dependency, and computationally expensive bird’s-eye view (BEV) representation. To address these limitations, we propose PRIX—the first monocular end-to-end planning framework operating directly on raw camera pixels, eliminating both LiDAR sensors and explicit BEV modeling. Its core innovation is the Context-aware Re-calibration Transformer (CaRT), which enhances visual representation through multi-scale feature re-calibration. Integrated with a lightweight vision encoder and a generative trajectory prediction head, PRIX achieves high planning accuracy within an efficient, compact architecture. Evaluated on NavSim and nuScenes, PRIX sets new state-of-the-art performance while offering faster inference speed and significantly fewer parameters than large multimodal diffusion-based models. Its sensor-minimal design and computational efficiency demonstrate strong practical viability for real-world deployment.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Large Vision Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

While end-to-end autonomous driving models show promising results, their practical deployment is often hindered by large model sizes, a reliance on expensive LiDAR sensors and computationally intensive BEV feature representations. This limits their scalability, especially for mass-market vehicles equipped only with cameras. To address these challenges, we propose PRIX (Plan from Raw Pixels). Our novel and efficient end-to-end driving architecture operates using only camera data, without explicit BEV representation and forgoing the need for LiDAR. PRIX leverages a visual feature extractor coupled with a generative planning head to predict safe trajectories from raw pixel inputs directly. A core component of our architecture is the Context-aware Recalibration Transformer (CaRT), a novel module designed to effectively enhance multi-level visual features for more robust planning. We demonstrate through comprehensive experiments that PRIX achieves state-of-the-art performance on the NavSim and nuScenes benchmarks, matching the capabilities of larger, multimodal diffusion planners while being significantly more efficient in terms of inference speed and model size, making it a practical solution for real-world deployment. Our work is open-source and the code will be at https://maxiuw.github.io/prix.

Problem

• Reducing reliance on LiDAR and BEV for autonomous driving
• Developing efficient end-to-end driving with only camera data
• Enhancing visual features for robust trajectory planning

Innovation

• Uses only camera data for autonomous driving
• Eliminates need for LiDAR and BEV representation
• Introduces Context-aware Recalibration Transformer (CaRT)