Evaluation of adaptive sampling methods in scenario generation for virtual safety impact assessment of pre-crash safety systems

2025-03-02

AI Summary

High-dimensional scenario parameters in virtual safety assessment for ADAS/ADS cause combinatorial explosion, severely hindering efficient and reliable evaluation. Method: This paper systematically compares importance sampling and active sampling under adaptive sample space reduction (ASSR), stratified sampling, and batched simulation. Contribution/Results: We quantitatively demonstrate— for the first time—that ASSR reduces active sampling estimation error by up to 90%; that combining ASSR with stratification eliminates performance gaps between the two sampling methods; and that large batch sizes significantly reduce wall-clock time at the cost of increased simulator invocations. Based on these findings, we propose a principled sampling configuration guideline balancing accuracy and efficiency. The framework is scalable, reproducible, and specifically designed for high-dimensional scenario generation in virtual safety validation.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Multiagent Systems: Adversarial Agents
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Virtual safety assessment plays a vital role in evaluating the safety impact of pre-crash safety systems such as advanced driver assistance systems (ADAS) and automated driving systems (ADS). However, as the number of parameters in simulation-based scenario generation increases, the number of crash scenarios to simulate grows exponentially, making complete enumeration computationally infeasible. Efficient sampling methods, such as importance sampling and active sampling, have been proposed to address this challenge. However, a comprehensive evaluation of how domain knowledge, stratification, and batch sampling affect their efficiency remains limited. This study evaluates the performance of importance sampling and active sampling in scenario generation, incorporating two domain-knowledge-driven features: adaptive sample space reduction (ASSR) and stratification. Additionally, we assess the effects of a third feature, batch sampling, on computational efficiency in terms of both CPU and wall-clock time. Based on our findings, we provide practical recommendations for applying ASSR, stratification, and batch sampling to optimize sampling performance. Our results demonstrate that ASSR substantially improves sampling efficiency for both importance sampling and active sampling. When integrated into active sampling, ASSR reduces the root mean squared estimation error (RMSE) of the estimates by up to 90%. Stratification further improves sampling performance for both methods, regardless of ASSR implementation. When ASSR and/or stratification are applied, importance sampling performs on par with active sampling, whereas when neither feature is used, active sampling is more efficient. Larger batch sizes reduce wall-clock time but increase the number of simulations required to achieve the same estimation accuracy.

Problem

• Evaluates adaptive sampling methods for virtual safety impact assessment.
• Assesses efficiency of importance and active sampling with domain knowledge.
• Provides recommendations for optimizing sampling performance in crash scenarios.

Innovation

• Adaptive sample space reduction improves efficiency.
• Stratification enhances sampling performance significantly.
• Batch sampling reduces wall-clock time effectively.