TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations

2026-06-15

AI Summary

This work proposes a decoupled approach to end-to-end autonomous driving that circumvents reliance on expensive expert demonstrations or high-cost image-based reinforcement learning in closed-loop settings. The method first pre-trains a driving policy via self-play reinforcement learning in a vectorized simulation environment, generating diverse and challenging edge-case scenarios. Subsequently, using unlabeled paired data of images and corresponding scene states, it aligns the policy’s latent space with a pretrained vision backbone through a combination of action KL divergence and a batch-wise relational low-rank structural loss. This alignment enables fully end-to-end training without expert supervision. Evaluated in a realistic closed-loop environment built upon 3D Gaussian splatting, the proposed approach achieves performance on par with or superior to current state-of-the-art methods.

Abstract

End-to-end autonomous driving has achieved state-of-the-art performance on benchmarks and real-world deployments. Its standard training recipe, however, is expensive across all stages: collecting and labeling millions of driving frames is costly, and closed-loop RL on images is bottlenecked by the per-step cost of photorealistic rendering plus a forward pass through a large vision backbone. Self-play in vectorized simulators changes the economics: millions of rollout steps per second, and a state distribution naturally rich in collisions, near-misses, and recoveries that no driving log contains. Our approach exploits this asymmetry by decoupling learning to drive from learning to see. We pretrain a single policy by self-play, then align its latent space with a pretrained vision backbone, through the action KL divergence and a batch-relational low-rank structural loss. The action target comes from the self-play policy, so alignment never supervises against a logged trajectory: a paired dataset of (image, scene-state) frames suffices, with no need for the curated expert demonstrations that imitation pretraining is built on. On photorealistic 3D Gaussian splatting closed-loop scenarios, the resulting end-to-end policy matches or exceeds prior end-to-end methods.

Problem

• end-to-end autonomous driving
• expert demonstrations
• imitation learning
• closed-loop RL
• data efficiency

Innovation

• self-play
• decoupled perception and control
• expert-free imitation
• latent alignment
• vectorized simulation