LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulation

2025-04-15

AI Summary

To address the challenge of flexibly generating high-fidelity, safety-critical traffic scenarios from natural language for scalable autonomous driving testing, this paper proposes the first language-driven multi-agent trajectory generation framework. Methodologically, we design a language-conditioned diffusion model coupled with a closed-loop training strategy, construct a large-scale interactive language-annotated dataset—Inter-Drive—and integrate a CLIP encoder, interaction-aware trajectory modeling, and an end-to-end generation pipeline. Our contributions are threefold: (1) the first framework enabling fine-grained, safety-constrained, joint behavioral generation for multiple agents conditioned on natural language; (2) empirical validation on the Waymo Motion dataset demonstrating superior realism, language alignment, and counterfactual scenario generation capability; and (3) significant improvements in testing efficiency and controllability, establishing a novel paradigm for closed-loop verification of autonomous driving systems.

Technology Category

• Natural Language Processing: Generation
• Planning, Routing, and Scheduling: Planning with Language Models
• Multiagent Systems: Adversarial Agents

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Evaluating autonomous vehicles with controllability enables scalable testing in counterfactual or structured settings, enhancing both efficiency and safety. We introduce LangTraj, a language-conditioned scene-diffusion model that simulates the joint behavior of all agents in traffic scenarios. By conditioning on natural language inputs, LangTraj provides flexible and intuitive control over interactive behaviors, generating nuanced and realistic scenarios. Unlike prior approaches that depend on domain-specific guidance functions, LangTraj incorporates language conditioning during training, facilitating more intuitive traffic simulation control. We propose a novel closed-loop training strategy for diffusion models, explicitly tailored to enhance stability and realism during closed-loop simulation. To support language-conditioned simulation, we develop Inter-Drive, a large-scale dataset with diverse and interactive labels for training language-conditioned diffusion models. Our dataset is built upon a scalable pipeline for annotating agent-agent interactions and single-agent behaviors, ensuring rich and varied supervision. Validated on the Waymo Motion Dataset, LangTraj demonstrates strong performance in realism, language controllability, and language-conditioned safety-critical simulation, establishing a new paradigm for flexible and scalable autonomous vehicle testing.

Problem

• Simulate traffic scenarios with language-conditioned control
• Enhance autonomous vehicle testing via realistic behavior simulation
• Develop dataset for training language-guided diffusion models

Innovation

• Language-conditioned diffusion model for traffic simulation
• Closed-loop training strategy enhancing stability and realism
• Large-scale dataset with diverse interactive behavior labels