Learning Rollout from Sampling:An R1-Style Tokenized Traffic Simulation Model

2026-03-25

AI Summary

This work proposes R1Sim, a novel traffic simulation framework that addresses the limited diversity and fidelity of multi-agent driving behaviors in existing next-token prediction approaches, which often fail to actively explore highly uncertain motion tokens. R1Sim introduces, for the first time, an entropy-driven exploration mechanism that adaptively samples high-potential behaviors guided by motion token entropy. To balance exploration and exploitation, it integrates a safety-aware reward function with Group Relative Policy Optimization (GRPO). Evaluated on the Waymo Sim Agents benchmark, R1Sim achieves state-of-the-art performance, significantly enhancing both behavioral realism and diversity while maintaining stringent safety guarantees.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Learning diverse and high-fidelity traffic simulations from human driving demonstrations is crucial for autonomous driving evaluation. The recent next-token prediction (NTP) paradigm, widely adopted in large language models (LLMs), has been applied to traffic simulation and achieves iterative improvements via supervised fine-tuning (SFT). However, such methods limit active exploration of potentially valuable motion tokens, particularly in suboptimal regions. Entropy patterns provide a promising perspective for enabling exploration driven by motion token uncertainty. Motivated by this insight, we propose a novel tokenized traffic simulation policy, R1Sim, which represents an initial attempt to explore reinforcement learning based on motion token entropy patterns, and systematically analyzes the impact of different motion tokens on simulation outcomes. Specifically, we introduce an entropy-guided adaptive sampling mechanism that focuses on previously overlooked motion tokens with high uncertainty yet high potential. We further optimize motion behaviors using Group Relative Policy Optimization (GRPO), guided by a safety-aware reward design. Overall, these components enable a balanced exploration-exploitation trade-off through diverse high-uncertainty sampling and group-wise comparative estimation, resulting in realistic, safe, and diverse multi-agent behaviors. Extensive experiments on the Waymo Sim Agent benchmark demonstrate that R1Sim achieves competitive performance compared to state-of-the-art methods.

Problem

• traffic simulation
• motion token
• exploration-exploitation
• autonomous driving evaluation
• behavior diversity

Innovation

• entropy-guided sampling
• tokenized traffic simulation
• reinforcement learning
• Group Relative Policy Optimization
• multi-agent behavior generation