See Less, Drive Better: Generalizable End-to-End Autonomous Driving via Foundation Models Stochastic Patch Selection
This work addresses the limited out-of-distribution (OOD) generalization of end-to-end autonomous driving policies, which stems from the high redundancy in patch-level features extracted by foundation models, leading policies to overfit spurious correlations. For the first time, this study identifies and quantifies such redundancy and introduces Stochastic Patch Selection (SPS), a mechanism that randomly masks a subset of image patch features per frame while preserving spatial layout, thereby compelling the policy to learn invariant decision cues from diverse yet complete scene views. Leveraging features from foundation models such as BLIP2, combined with PCA and cross-patch similarity analysis, SPS improves OOD performance by 6.2% on average—and up to 20.4%—in closed-loop simulation, accelerates inference by 2.4×, and enables deployment on real vehicles without fine-tuning, outperforming existing state-of-the-art methods in eight out of nine ablation settings.