TerraZero: Procedural Driving Simulation for Zero-Demonstration Self-Play at Scale

2026-07-14

AI Summary

Training robust autonomous driving agents requires balancing simulation efficiency, realism, and coverage of long-tail scenarios. This work proposes a pure reinforcement learning framework that relies solely on real-world map geometry and procedurally generates traffic participants, traffic signals, vehicle dynamics, and reward functions, enabling end-to-end training without human demonstrations or fallback planners. The method achieves, for the first time, a fully learned policy that operates without demonstrations or fallback mechanisms, demonstrating zero-shot generalization across multiple cities and handling complex traffic rules such as left- and right-hand driving. Leveraging a CPU/GPU heterogeneous architecture, the system attains high training throughput of 1.3 million agent steps per second. It outperforms existing demonstration-free approaches on the InterPlan long-tail benchmark, the val14 safety test, and the Waymo Open Sim Agents challenge, matching the performance of the strongest reference-anchor self-play models.

Abstract

Training robust autonomous driving agents requires a simulator that is fast enough for reinforcement learning at scale, realistic enough to ground behavior in real-world map structure, and diverse enough to cover the safety-critical long tail that logged data rarely contains. We present TerraZero, a procedural driving simulator and self-play training stack. A configurable C engine runs simulation on the CPU and policy inference on the GPU over a zero-copy path, sustaining 1.3M agent-steps per second on a single server-grade GPU, far faster than existing object-level simulators, while keeping fidelity lighter single-agent systems omit: heterogeneous agents, multiple dynamics models, and full traffic-rule enforcement. TerraZero treats logged data only as a source of real-world map geometry, populating each map with randomized rule-based road users and signal controllers and randomizing agent dynamics, rewards, and sizes per episode, so a map yields an unbounded set of scenarios. Every reported policy trains from scratch by reinforcement learning alone on a compute-efficient self-play recipe across GPUs, with zero human demonstrations and no fallback planner at inference. Policies generalize zero-shot across cities and datasets, including emergent left-hand-traffic driving without explicit supervision. As an ego policy, TerraZero is the first fully learned policy to top the InterPlan long-tail benchmark, ahead of larger learned planners; on routine-driving val14 it ranks among the best approaches and is the safest, posting the best collision and time-to-collision scores. On Waymo Open Sim Agents realism the same recipe outperforms other demonstration-free methods and is competitive with the strongest reference-anchored self-play method. One stack serves both roles: driving policies across dynamics for cars and trucks, and sim agents that jointly control vehicles, pedestrians, and cyclists.

Problem

• autonomous driving
• zero-demonstration
• procedural simulation
• long-tail scenarios
• self-play

Innovation

• procedural simulation
• zero-demonstration reinforcement learning
• self-play at scale
• heterogeneous traffic agents
• zero-shot generalization