Video2Track: From Real-World Interaction Videos to Steerable Adversarial Closed-Track Testing for Automated Driving Systems

2026-08-11

AI Summary

This work addresses the limitation of existing closed-course testing, which relies on standardized protocols or predefined trajectories and fails to capture the complexity of real-world road interactions. The authors propose an end-to-end framework that integrates video semantic mapping and dynamic interaction testing to generate controllable adversarial scenarios from real driving videos, enabling adjustable risk levels and interaction styles for the first time. By combining vision-language models, retrieval-augmented generation, conditional diffusion models, and Stackelberg game theory, the method supports high-fidelity and scalable validation of autonomous driving systems. Experimental results demonstrate that the approach successfully reproduces and diversifies real-world interactive scenarios in closed-course environments, significantly enhancing the realism and flexibility of autonomous vehicle testing.

Abstract

Closed-track testing plays a fundamental role in the verification and validation of automated driving systems (ADS), particularly for safety-critical scenarios, by enabling reproducible evaluation under controlled conditions. However, most existing approaches still rely on standardized protocols or predefined trajectories, leading to overly scripted interactions and limited ability to reproduce the natural complexity of public-road traffic. To address this limitation, we propose Video2Track, a framework that transfers real-world interactive driving scenarios from videos into steerable adversarial closed-track testing. The framework consists of two tightly coupled modules. The first is a scenario semantic mapping module, which extracts structured semantics from driving videos using a vision-language model and grounds them onto a closed-track topology library via retrieval-augmented generation, thereby identifying compatible map segments and interaction anchors. The second is a dynamic interactive testing module, which conditions on the grounded topology and anchors to generate diverse multi-agent trajectories through a conditional diffusion model, while regulating interaction intensity via a Stackelberg game with a parameterized adversarial objective. Closed-track experiments demonstrate that the proposed framework can faithfully reproduce representative real-world interaction scenarios and generate executable scenario variants with controllable risk levels and interaction styles, providing a scalable approach for realistic and steerable ADS validation.

Problem

• closed-track testing
• automated driving systems
• real-world interaction
• scenario reproduction
• traffic complexity

Innovation

• closed-track testing
• video-to-scenario generation
• conditional diffusion model
• adversarial interaction
• retrieval-augmented generation