LLM-based Realistic Safety-Critical Driving Video Generation

2025-07-01

AI Summary

Evaluating autonomous driving systems requires realistic, diverse, and efficiently generated safety-critical edge-case scenarios (e.g., occluded pedestrians crossing, sudden cut-ins), yet existing approaches struggle to simultaneously satisfy fidelity, variability, and scalability. Method: This paper proposes an end-to-end driving scenario video generation framework powered by large language models (LLMs). It employs few-shot prompting to elicit structured safety-event scripts from LLMs; integrates CARLA for high-fidelity physics simulation and multi-agent coordination; and leverages Cosmos-Transfer1 and ControlNet to build a controllable rendering pipeline that transforms synthetic images into photorealistic videos. Contribution/Results: Experiments demonstrate that the framework automatically generates diverse, high-fidelity video sequences covering rare hazardous scenarios. It significantly improves simulation-to-reality transfer capability and enhances the robustness evaluation efficacy of autonomous driving systems, enabling more rigorous and scalable safety validation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Designing diverse and safety-critical driving scenarios is essential for evaluating autonomous driving systems. In this paper, we propose a novel framework that leverages Large Language Models (LLMs) for few-shot code generation to automatically synthesize driving scenarios within the CARLA simulator, which has flexibility in scenario scripting, efficient code-based control of traffic participants, and enforcement of realistic physical dynamics. Given a few example prompts and code samples, the LLM generates safety-critical scenario scripts that specify the behavior and placement of traffic participants, with a particular focus on collision events. To bridge the gap between simulation and real-world appearance, we integrate a video generation pipeline using Cosmos-Transfer1 with ControlNet, which converts rendered scenes into realistic driving videos. Our approach enables controllable scenario generation and facilitates the creation of rare but critical edge cases, such as pedestrian crossings under occlusion or sudden vehicle cut-ins. Experimental results demonstrate the effectiveness of our method in generating a wide range of realistic, diverse, and safety-critical scenarios, offering a promising tool for simulation-based testing of autonomous vehicles.

Problem

• Generating diverse safety-critical driving scenarios for autonomous vehicle testing
• Bridging simulation-reality gap with realistic video generation techniques
• Automating rare edge case creation via LLM-based code synthesis

Innovation

• LLM-based few-shot code generation for scenarios
• Cosmos-Transfer1 with ControlNet for realism
• Automated safety-critical edge case creation