Agent-driven Long-tail Simulation for Autonomous Driving

2026-07-05

AI Summary

Current autonomous driving simulation systems struggle to generate semantically rich and behaviorally diverse long-tail interactive scenarios. This work proposes the first closed-loop simulation framework based on instruction-following large language models, which controls traffic participants through a structured action interface to produce realistic behaviors characterized by intentionality and reactivity. Building upon nuPlan, the authors introduce SemanticPlan, a new benchmark featuring multi-agent language instructions. Experimental results demonstrate that state-of-the-art motion planners perform poorly on this benchmark, confirming its challenging nature and highlighting its potential to guide future algorithmic improvements.

Abstract

Evaluating autonomous driving systems in closed-loop settings requires realistic and interactive simulation, yet existing simulators largely rely on log replay or rule-based agents, limiting behavioral diversity and long-tail coverage. We propose an agent-driven simulation framework in which surrounding road participants are controlled by instruction-following large language models through a structured action interface, enabling intentional and reactive behaviors while preserving physical plausibility. Furthermore, we introduce SemanticPlan, a benchmark of closed-loop planning in long-tail and semantically rich scenarios that augment real nuPlan scenes with multiple interactive agents following diverse language instructions. Evaluation results show that state-of-the-art planners still struggle to consistently achieve safe and effective task completion, suggesting that these long-tail scenarios remain challenging.

Problem

• long-tail simulation
• autonomous driving
• closed-loop evaluation
• behavioral diversity
• interactive agents

Innovation

• agent-driven simulation
• large language models
• long-tail scenarios
• closed-loop planning
• SemanticPlan