Safety Evaluation of Motion Plans Using Trajectory Predictors as Forward Reachable Set Estimators

2025-07-30

AI Summary

This paper addresses the fundamental trade-off between completeness (i.e., zero false negatives in detecting unsafe motion plans) and reliability (i.e., zero false positives in labeling safe plans) in safety monitoring for end-to-end autonomous driving. To resolve this, we propose a forward reachable set estimation and verification framework grounded in trajectory prediction. Methodologically, we leverage a multimodal trajectory predictor to model the forward state distribution of surrounding agents; extract data-driven, convex reachable regions via convex optimization; ensure statistical coverage guarantees using conformal prediction; and dynamically calibrate conservatism against distribution shift via a Bayesian filter. Conditional modeling incorporates lane topology and agent history. Evaluated on nuScenes, our approach achieves >99% completeness while significantly improving the reasonableness of safety judgments. It establishes a verifiable, adaptive safety monitoring paradigm for learning-based autonomous driving systems.

Technology Category

• Natural Language Processing: Safety and Robustness
• Intelligent Robots: State Estimation
• Planning, Routing, and Scheduling: Model-Based Reasoning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

The advent of end-to-end autonomy stacks - often lacking interpretable intermediate modules - has placed an increased burden on ensuring that the final output, i.e., the motion plan, is safe in order to validate the safety of the entire stack. This requires a safety monitor that is both complete (able to detect all unsafe plans) and sound (does not flag safe plans). In this work, we propose a principled safety monitor that leverages modern multi-modal trajectory predictors to approximate forward reachable sets (FRS) of surrounding agents. By formulating a convex program, we efficiently extract these data-driven FRSs directly from the predicted state distributions, conditioned on scene context such as lane topology and agent history. To ensure completeness, we leverage conformal prediction to calibrate the FRS and guarantee coverage of ground-truth trajectories with high probability. To preserve soundness in out-of-distribution (OOD) scenarios or under predictor failure, we introduce a Bayesian filter that dynamically adjusts the FRS conservativeness based on the predictor's observed performance. We then assess the safety of the ego vehicle's motion plan by checking for intersections with these calibrated FRSs, ensuring the plan remains collision-free under plausible future behaviors of others. Extensive experiments on the nuScenes dataset show our approach significantly improves soundness while maintaining completeness, offering a practical and reliable safety monitor for learned autonomy stacks.

Problem

• Ensuring motion plan safety in end-to-end autonomy stacks
• Calibrating trajectory predictors for complete and sound safety monitoring
• Detecting collision risks using data-driven forward reachable sets

Innovation

• Leverage trajectory predictors for reachable sets
• Use conformal prediction to calibrate coverage
• Dynamic Bayesian filter adjusts conservativeness