InfGen: Scenario Generation as Next Token Group Prediction

2025-06-29

AI Summary

Existing data-driven traffic simulation methods predominantly rely on static initialization or log replay, limiting their ability to support dynamically evolving, long-horizon, and interactive scenarios with variable agent composition. This paper introduces the first autoregressive generative framework that models traffic scenes as unified token sequences. Leveraging a Transformer architecture, it jointly tokenizes and end-to-end models traffic signals, agent states, and motion vectors. The framework enables real-time agent insertion and removal, supports arbitrarily long temporal horizons, and delivers high-fidelity interactive simulation. Experimental results demonstrate substantial improvements in scene realism and diversity. Reinforcement learning policies trained within this simulated environment exhibit enhanced robustness and generalization, validating the framework’s effectiveness and novelty as a high-fidelity simulation platform for next-generation autonomous driving training and evaluation.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Multiagent Systems: Adversarial Agents
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Realistic and interactive traffic simulation is essential for training and evaluating autonomous driving systems. However, most existing data-driven simulation methods rely on static initialization or log-replay data, limiting their ability to model dynamic, long-horizon scenarios with evolving agent populations. We propose InfGen, a scenario generation framework that outputs agent states and trajectories in an autoregressive manner. InfGen represents the entire scene as a sequence of tokens, including traffic light signals, agent states, and motion vectors, and uses a transformer model to simulate traffic over time. This design enables InfGen to continuously insert new agents into traffic, supporting infinite scene generation. Experiments demonstrate that InfGen produces realistic, diverse, and adaptive traffic behaviors. Furthermore, reinforcement learning policies trained in InfGen-generated scenarios achieve superior robustness and generalization, validating its utility as a high-fidelity simulation environment for autonomous driving. More information is available at https://metadriverse.github.io/infgen/.

Problem

• Generates dynamic traffic scenarios for autonomous driving
• Overcomes limitations of static or replay-based simulation methods
• Enables infinite scene generation with new agent insertion

Innovation

• Autoregressive agent state and trajectory generation
• Token-based scene representation with transformers
• Continuous agent insertion for infinite scenes