Perception in Plan: Coupled Perception and Planning for End-to-End Autonomous Driving

2025-08-15

AI Summary

To address the loose coupling between perception and planning, and the lack of task-oriented guidance in perception within end-to-end autonomous driving, this paper proposes a “Perception-in-Planning” framework. The method introduces multimodal anchor trajectories as planning priors to dynamically steer perception toward critical traffic entities along expected paths. It jointly optimizes a differentiable perception module and an autoregressive trajectory predictor, enabling planning-driven dynamic attention. This facilitates tight perception–planning co-modeling and end-to-end joint training. Evaluated on NAVSIM and Bench2Drive benchmarks, the approach achieves state-of-the-art performance, significantly improving trajectory prediction accuracy and decision-making reliability. Key contributions include: (1) a novel planning-guided perception paradigm; (2) differentiable, co-optimized perception and planning modules; and (3) empirical validation of enhanced robustness and generalization in complex urban driving scenarios.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Planning with Language Models
• Multiagent Systems: Multiagent Planning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

End-to-end autonomous driving has achieved remarkable advancements in recent years. Existing methods primarily follow a perception-planning paradigm, where perception and planning are executed sequentially within a fully differentiable framework for planning-oriented optimization. We further advance this paradigm through a perception-in-plan framework design, which integrates perception into the planning process. This design facilitates targeted perception guided by evolving planning objectives over time, ultimately enhancing planning performance. Building on this insight, we introduce VeteranAD, a coupled perception and planning framework for end-to-end autonomous driving. By incorporating multi-mode anchored trajectories as planning priors, the perception module is specifically designed to gather traffic elements along these trajectories, enabling comprehensive and targeted perception. Planning trajectories are then generated based on both the perception results and the planning priors. To make perception fully serve planning, we adopt an autoregressive strategy that progressively predicts future trajectories while focusing on relevant regions for targeted perception at each step. With this simple yet effective design, VeteranAD fully unleashes the potential of planning-oriented end-to-end methods, leading to more accurate and reliable driving behavior. Extensive experiments on the NAVSIM and Bench2Drive datasets demonstrate that our VeteranAD achieves state-of-the-art performance.

Problem

• Integrates perception into planning for autonomous driving
• Enhances planning with targeted perception guided by objectives
• Uses multi-mode trajectories to improve perception and planning

Innovation

• Perception-in-plan framework integrates perception with planning
• Multi-mode anchored trajectories guide targeted perception
• Autoregressive strategy predicts future trajectories progressively