ECoSim: Data Efficient Fine-Tuning for Controllable Traffic Simulation

2026-07-01

AI Summary

This work addresses the inefficiency of existing controllable traffic simulation methods, which rely on extensive labeled data to retrain large generative models and struggle to support multimodal control. The authors propose a lightweight control adaptation framework that modulates intermediate features of pretrained diffusion and autoregressive traffic models via identity-initialized FiLM layers. Requiring less than 1% of paired control data, this approach enables precise responses to diverse multimodal inputs—including sketches, behavior codes, and textual prompts—without retraining the backbone models. The method substantially improves data efficiency and scene diversity, facilitates counterfactual reasoning and synthesis of long-tail scenarios, and demonstrates high-fidelity, safe, and controllable closed-loop driving simulation capabilities, as validated in the Waymo Open Sim Agents Challenge.

Abstract

Controllable traffic simulation is critical for testing autonomous driving systems, yet existing approaches often require retraining large generative models with extensive annotated data. We introduce a lightweight control adaptation framework that enables multi-modal controllability (sketch, latent behavior codes, and text) for pretrained state-of-the-art diffusion and autoregressive traffic models. By modulating intermediate features through identity-initialized FiLM layers, our method efficiently adds new control modalities while preserving the base model's generative prior. Evaluated on Waymo Open Sim Agents Challenge, our approach demonstrates strong controllability with less than 1% of the paired control data. Through context-aware condition transfer, our framework enables counterfactual scenario generation and long-tail synthesis while maintaining stable closed-loop driving realism and safety. Our framework unlocks new possibilities for controllable traffic simulation, enabling targeted scenario generation through lightweight adaptation of pretrained generative models. Project page: https://ecosim-web.github.io/

Problem

• controllable traffic simulation
• data efficiency
• generative models
• autonomous driving testing
• multi-modal control

Innovation

• controllable simulation
• data-efficient fine-tuning
• multi-modal control
• FiLM modulation
• pretrained generative models