FAST: A Framework for Aligned Sampling and Training in Parallel Reinforcement Learning for Autonomous Driving

2026-06-19

AI Summary

This work addresses the inefficiency in parallel reinforcement learning for autonomous driving caused by the “straggler effect,” where early termination of a single environment forces synchronized resets, leading to wasted samples and high latency. The authors propose FAST, a novel framework that introduces the first termination-aware dynamic synchronization mechanism. It employs Dynamic Parallel Sampling Alignment (DPSA) to extend terminated trajectories, preserving vectorized execution while adaptively triggering global truncation based on termination rates. Combined with Scaled Masked Policy Optimization (SMPO), validity masking, and adaptive loss normalization, FAST avoids premature resets without introducing statistical bias, thereby maintaining both data diversity and theoretical consistency. Experiments demonstrate that FAST achieves at least a 1.78× wall-clock speedup over single-trajectory baselines.

Abstract

Deep reinforcement learning is pivotal for closed-loop autonomous driving yet remains constrained by severe bottlenecks in sampling efficiency. Standard parallel sampling mitigates this but suffers from the straggler effect, where the premature termination of a single environment necessitates a synchronized batch re-initialization, leading to suboptimal sample utilization and prohibitive re-initialization latency. To address this, we propose FAST, a synchronous parallel framework tailored for closed-loop simulation. Specifically, FAST employs Dynamic Parallel Sampling Alignment (DPSA) to maintain vectorization synchronization by extending terminated episodes via virtual continuation, thereby decoupling the sampling loop from individual terminations. By dynamically triggering global truncation based on the termination rate of parallel clips, FAST effectively eliminates the bottleneck of premature resets without sacrificing data diversity. Furthermore, to strictly preserve theoretical consistency, we incorporate a Scaled Mask-Padding Optimization (SMPO) that leverages validity masking and adaptive loss normalization to nullify the bias from auxiliary padding data. Empirical evaluations demonstrate that FAST achieves at least a 1.78 times wall-clock speedup over the single-clip baseline while preserving statistical unbiasedness.

Problem

• straggler effect
• sampling efficiency
• parallel reinforcement learning
• episode termination
• synchronous sampling

Innovation

• Dynamic Parallel Sampling Alignment
• Scaled Mask-Padding Optimization
• straggler effect mitigation
• parallel reinforcement learning
• closed-loop autonomous driving