Decictor: Towards Evaluating the Robustness of Decision-Making in Autonomous Driving Systems

2024-02-28

AI Summary

Robustness evaluation of autonomous driving system (ADS) path planning and decision-making (PPD) under environmental perturbations suffers from three key challenges: absence of effective assessment methodologies, lack of well-defined optimality criteria, and intractability of high-dimensional scenario search. Method: This paper proposes, for the first time, an automated non-optimal decision scenario (NoDS) generation framework. It integrates non-intrusive environmental mutation, trajectory consistency verification, and spatiotemporal coupled feedback control to overcome the dual bottlenecks of implicit PPD optimality criteria and combinatorial explosion in scenario space. Contribution/Results: Evaluated on the Baidu Apollo platform using production-grade simulation, the method successfully identifies diverse PPD robustness deficiencies. It significantly improves detection efficiency and interpretability of suboptimal path planning behaviors, establishing a novel, scalable paradigm for trustworthy evaluation of industrial ADS deployments.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling
• Multiagent Systems: Multiagent Planning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Autonomous Driving System (ADS) testing is crucial in ADS development, with the current primary focus being on safety. However, the evaluation of non-safety-critical performance, particularly the ADS's ability to make optimal decisions and produce optimal paths for autonomous vehicles (AVs), is also vital to ensure the intelligence and reduce risks of AVs. Currently, there is little work dedicated to assessing the robustness of ADSs' path-planning decisions (PPDs), i.e., whether an ADS can maintain the optimal PPD after an insignificant change in the environment. The key challenges include the lack of clear oracles for assessing PPD optimality and the difficulty in searching for scenarios that lead to non-optimal PPDs. To fill this gap, in this paper, we focus on evaluating the robustness of ADSs' PPDs and propose the first method, Decictor, for generating non-optimal decision scenarios (NoDSs), where the ADS does not plan optimal paths for AVs. Decictor comprises three main components: Non-invasive Mutation, Consistency Check, and Feedback. To overcome the oracle challenge, Non-invasive Mutation is devised to implement conservative modifications, ensuring the preservation of the original optimal path in the mutated scenarios. Subsequently, the Consistency Check is applied to determine the presence of non-optimal PPDs by comparing the driving paths in the original and mutated scenarios. To deal with the challenge of large environment space, we design Feedback metrics that integrate spatial and temporal dimensions of the AV's movement. These metrics are crucial for effectively steering the generation of NoDSs. We evaluate Decictor on Baidu Apollo, an open-source and production-grade ADS. The experimental results validate the effectiveness of Decictor in detecting non-optimal PPDs of ADSs.

Problem

• Autonomous Vehicles
• Optimal Route Selection
• Testing Methodologies

Innovation

• Decictor
• Autonomous Vehicles Route Optimization
• Environmental Adaptability Assessment