LD-Scene: LLM-Guided Diffusion for Controllable Generation of Adversarial Safety-Critical Driving Scenarios

2025-05-16

AI Summary

To address the challenges of evaluating autonomous driving systems in rare, high-risk scenarios—namely, poor controllability, low interpretability, and heavy reliance on expert knowledge—this paper proposes a natural-language-driven adversarial driving scenario generation framework. Methodologically, it integrates large language models (LLMs) with latent diffusion models (LDMs) to construct an end-to-end “instruction-to-adversarial-trajectory” generation pipeline. It introduces, for the first time, an LLM-guided interpretable adversarial loss mechanism, augmented by chain-of-thought (CoT)-based code generation and debugging modules, enabling precise semantic alignment and fine-grained behavioral control. Evaluated on the nuScenes dataset, the framework achieves state-of-the-art performance: generated scenarios exhibit high realism, diversity, and robustness, significantly enhancing both the effectiveness and interpretability of targeted stress testing for autonomous driving systems.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Ensuring the safety and robustness of autonomous driving systems necessitates a comprehensive evaluation in safety-critical scenarios. However, these safety-critical scenarios are rare and difficult to collect from real-world driving data, posing significant challenges to effectively assessing the performance of autonomous vehicles. Typical existing methods often suffer from limited controllability and lack user-friendliness, as extensive expert knowledge is essentially required. To address these challenges, we propose LD-Scene, a novel framework that integrates Large Language Models (LLMs) with Latent Diffusion Models (LDMs) for user-controllable adversarial scenario generation through natural language. Our approach comprises an LDM that captures realistic driving trajectory distributions and an LLM-based guidance module that translates user queries into adversarial loss functions, facilitating the generation of scenarios aligned with user queries. The guidance module integrates an LLM-based Chain-of-Thought (CoT) code generator and an LLM-based code debugger, enhancing the controllability and robustness in generating guidance functions. Extensive experiments conducted on the nuScenes dataset demonstrate that LD-Scene achieves state-of-the-art performance in generating realistic, diverse, and effective adversarial scenarios. Furthermore, our framework provides fine-grained control over adversarial behaviors, thereby facilitating more effective testing tailored to specific driving scenarios.

Problem

• Generating rare safety-critical driving scenarios for autonomous vehicles
• Overcoming limited controllability in existing adversarial scenario methods
• Enabling user-friendly natural language control for scenario generation

Innovation

• LLM-guided diffusion for adversarial scenario generation
• Natural language user queries to adversarial loss
• Chain-of-Thought code generator for controllability