Scenario Generation for Testing of Autonomous Driving Systems Using Real-World Failure Records

2026-06-30

AI Summary

This work addresses the limitations of existing autonomous driving testing methods, which rely on handcrafted templates or fixed models and struggle to efficiently reproduce complex real-world failure scenarios. The authors propose a modular synthesis framework powered by large language models (LLMs) that, for the first time, incorporates natural language descriptions and contextual information from real accident reports into the scenario generation process. Operating under predefined testing constraints, the framework automatically constructs diverse driving scenarios. Integrated with the MetaDrive platform and validated using NHTSA crash data, the approach successfully generates test cases covering four road types, three non-ego vehicle motion patterns, and construction-zone anomalies. Remarkably, only 20 generated scenarios effectively expose latent system failures, significantly outperforming conventional testing methodologies.

Abstract

To ensure safe on-road behavior, pre-deployment testing and failure discovery of Autonomous Driving Systems (ADS) is crucial. Present day simulation based testing methods focus largely on mathematical models for efficient search of optimal scenarios, assuming a fixed scenario representation. On the other hand, real-world testing involves substantial manual effort to design scenario templates for testing. These templates represent distinct failure scenarios consisting of pre-deployment vehicle movements, map types, etc. Historical failure records for ADS are a reliable source of real-world failure conditions, which can be used for scenario generation. In this work, we propose a scenario generation pipeline using categorical and contextual information available from historical records in natural language format. Our approach consists of modular LLM based synthetic scenario generation, compatible with the testing constraints of a given system. We successfully apply our method to generate a diverse set of scenarios for testing autonomous navigation on Metadrive simulator using the NHTSA ADS crash records. Our approach results in accurate and diverse scenario generation with a combination of 4 road types, 3 non ego vehicle movement types, including on road anomalies in the form of working zones. Generated scenarios align with the provided testing conditions, and reveals interesting failures of the system within a limited testing budget of 20 scenarios. Code is available at https://github.com/anjaliParashar/crash2scenario.

Problem

• Scenario Generation
• Autonomous Driving Systems
• Failure Records
• Simulation Testing
• Real-World Scenarios

Innovation

• scenario generation
• autonomous driving testing
• large language models
• failure records
• simulation-based validation