RCG: Safety-Critical Scenario Generation for Robust Autonomous Driving via Real-World Crash Grounding

2025-07-14

AI Summary

Safety-critical scenarios are extremely scarce in real-world driving data, severely limiting the robustness training and evaluation of autonomous driving systems. To address this, we propose a real-accident-semantic-guided adversarial scenario generation method: first, contrastive learning pretraining on large-scale driving logs is performed to learn general driving representations; then, few-shot fine-tuning on real crash video trajectories yields safety-aware behavioral embeddings. Subsequently, an embedding-space-driven adversarial trajectory selection mechanism generates high-risk, behaviorally realistic, and diverse opponent vehicle trajectories. Our approach seamlessly integrates into existing generative pipelines without architectural modifications. Evaluated across seven downstream assessment settings—including corner-case stress testing and cross-scenario generalization—it achieves an average 9.2% improvement in agent task success rate. The generated scenarios significantly enhance test authenticity, challenge level, and generalizability, enabling more rigorous and reliable evaluation of autonomous driving policies.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Adversarial Learning & Robustness
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Safety-critical scenarios are essential for training and evaluating autonomous driving (AD) systems, yet remain extremely rare in real-world driving datasets. To address this, we propose Real-world Crash Grounding (RCG), a scenario generation framework that integrates crash-informed semantics into adversarial perturbation pipelines. We construct a safety-aware behavior representation through contrastive pre-training on large-scale driving logs, followed by fine-tuning on a small, crash-rich dataset with approximate trajectory annotations extracted from video. This embedding captures semantic structure aligned with real-world accident behaviors and supports selection of adversary trajectories that are both high-risk and behaviorally realistic. We incorporate the resulting selection mechanism into two prior scenario generation pipelines, replacing their handcrafted scoring objectives with an embedding-based criterion. Experimental results show that ego agents trained against these generated scenarios achieve consistently higher downstream success rates, with an average improvement of 9.2% across seven evaluation settings. Qualitative and quantitative analyses further demonstrate that our approach produces more plausible and nuanced adversary behaviors, enabling more effective and realistic stress testing of AD systems. Code and tools will be released publicly.

Problem

• Generate safety-critical scenarios for autonomous driving training
• Integrate crash-informed semantics into adversarial perturbation pipelines
• Improve realism and effectiveness of stress testing AD systems

Innovation

• Integrates crash-informed semantics into adversarial pipelines
• Uses contrastive pre-training for safety-aware behavior representation
• Replaces handcrafted scoring with embedding-based criterion