CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving

2026-07-02

AI Summary

This work proposes an end-to-end autonomous driving policy based on large-scale closed-loop reinforcement learning to address the limitations of existing approaches that rely primarily on open-loop imitation learning and struggle with distribution shifts during closed-loop inference. By leveraging prior knowledge from Vision-Language-Action (VLA) models, the method learns a residual waypoint policy that effectively integrates semantic understanding with low-level control signals. Furthermore, a heterogeneous distributed simulation training architecture is introduced to enable efficient parallelized closed-loop training. The proposed approach achieves state-of-the-art performance on challenging benchmarks, including CARLA Longest6 v2 and Bench2Drive, significantly outperforming current methods.

Abstract

End-to-end autonomous driving (E2E-AD) aims to directly map raw sensor information to driving actions. Recently, with the rapid advancement of multi-modal large language models (MLLMs), researchers have proposed the paradigm of Vision-Language-Action (VLA) models for E2E-AD, where it seeks to integrate visual perception, language understanding and action prediction within a single policy. However, existing VLA-based policies largely adopts imitation learning, where it only learns to drive by optimizing distance-based metrics w.r.t. logged expert trajectories. Such distribution shift between open-loop training and closed-loop inference leads to suboptimal performance in closed-loop planning. To close this gap, we present CLEAR, a system that enables closed-loop training using Reinforcement Learning (RL) at scale for E2E-AD. We propose to learn a novel residual waypoint policy around the waypoint prior from pretrained VLA policies, effectively harnessing the knowledge within. On another front, one of the key challenges to scale up RL for vision-based policies is the number of parallel simulation environments since RL is data hungry. To that end, we design a heterogeneous pipeline that places the simulator and the VLA learner on distinct compute groups, which allows us to dramatically increase the number of simulation environments running in parallel while avoiding resource contention and maintaining training stability. We show that with a simple reward, CLEAR significantly outperforms previous methods and sets new state-of-the-art performance on the challenging benchmarks of CARLA longest6 v2 and Bench2Drive.

Problem

• end-to-end autonomous driving
• closed-loop planning
• distribution shift
• vision-language-action models
• imitation learning

Innovation

• Closed-loop Reinforcement Learning
• End-to-End Autonomous Driving
• Vision-Language-Action Models
• Residual Waypoint Policy
• Heterogeneous Simulation Pipeline