Closed Loop Dynamic Driving Data Mixture for Real-Synthetic Co-Training

2026-05-20

AI Summary

This work addresses the challenges in end-to-end autonomous driving where real-world data annotation is costly and exhibits scene bias, while indiscriminate use of synthetic data often leads to distribution shift. To this end, the authors propose AutoScale, a closed-loop data engine that formulates data mixing as a dynamic optimization problem driven by closed-loop performance feedback. AutoScale introduces Graph-RAE for unified scene representation, employs Cluster-GA for cluster-aware gradient reweighting, and integrates a cluster-guided vector retrieval mechanism to automatically optimize the ratio of real to synthetic data. Evaluated on the NavSim benchmark, AutoScale achieves significantly better performance than co-training and cross-domain baselines under constrained training budgets, using fewer synthetic samples.

Abstract

Data scaling is fundamental to modern deep learning, and grows increasingly critical as autonomous driving shifts to end-to-end learning. Real-world driving data is expensive to annotate and scene-biased, making real-synthetic co-training with near-infinite synthetic data a promising direction. However, naively incorporating all available synthetic data is inefficient and leads to distribution shifts, and optimizing data mixture under practical training budgets remains a critical yet under-explored problem. In this sense, we claim that the mixture of training data requires clear guidance in terms of scene types and quantities. Particularly in this work, we conceptualize the data mixture approximately as a dynamic optimization process that iteratively adjusts the training data mixture to maximize model performance, guided by closed-loop evaluation feedback, and propose AutoScale, a fully automated closed-loop data engine unifying scene representation, data mixture optimization and retrieval, as well as model training and evaluation. Specifically, we propose Graph Regularized AutoEncoder (Graph-RAE) for driving scene representations, introduce Cluster-aware Gradient Ascent (Cluster-GA) for cluster-wise importance estimation and reweighting, and perform cluster-guided vector retrieval to select high-value samples. Experiments on NavSim demonstrate that AutoScale outperforms vanilla co-training and cross-domain baselines, achieving better performance with fewer synthetic samples under constrained budgets.

Problem

• data mixture
• real-synthetic co-training
• autonomous driving
• distribution shift
• training budget

Innovation

• closed-loop data mixture
• real-synthetic co-training
• Graph Regularized AutoEncoder
• Cluster-aware Gradient Ascent
• dynamic data optimization