Beyond Imitation: Learning Safe End-to-End Autonomous Driving from Hard Negatives

2026-05-19

AI Summary

This work addresses a key limitation in existing end-to-end imitation learning for autonomous driving: the exclusive reliance on successful trajectories, which fails to distinguish geometrically similar yet safety-divergent driving behaviors. To overcome this, the authors propose BeyondDrive, a framework that jointly learns from both successful and failed driving behaviors to explicitly model safety boundaries. The approach introduces a flow-matching-based negative trajectory generator, a diversity-aware sampling strategy, and a repulsive distance loss, collectively capturing safety asymmetry and mitigating mode collapse. Evaluated on the NAVSIMv1 closed-loop benchmark, BeyondDrive achieves a PDMS score of 89.7, outperforming current state-of-the-art methods, and demonstrates strong generalization and zero-shot transfer capabilities across multimodal planners and the HUGSIM benchmark.

Abstract

Existing imitation learning methods for end-to-end autonomous driving predominantly learn from successful demonstrations by minimizing geometric deviations from expert trajectories. This paradigm implicitly assumes that spatial proximity implies behavioral safety, leading to a critical objective mismatch: trajectories with nearly identical imitation losses may exhibit drastically different safety outcomes, where one remains recoverable while the other results in collision. To address this limitation, we propose BeyondDrive, a failure-aware imitation learning framework that jointly learns from successful and failed driving behaviors. First, we introduce a flow matching-based negative trajectory generator that synthesizes safety-critical yet expert-proximate trajectories, enabling explicit modeling of safety asymmetry. Second, we develop a diversity-aware sampling strategy that mitigates mode collapse and improves coverage of diverse failure modes during negative trajectory generation. Third, we propose a Repulsive Distance Loss that simultaneously attracts predictions toward expert demonstrations while repelling them from hard negative trajectories, thereby establishing discriminative safety boundaries in trajectory space. Applied to the uni-modal baseline Latent TransFuser, BeyondDrive achieves 89.7 PDMS on the NAVSIMv1 closed-loop benchmark, outperforming prior state-of-the-art methods. Moreover, BeyondDrive generalizes effectively across different autonomous driving architectures, including multi-modal planners, and further demonstrates strong zero-shot transferability on the HUGSIM benchmark.

Problem

• imitation learning
• autonomous driving
• safety
• trajectory
• failure modes

Innovation

• failure-aware imitation learning
• hard negative trajectories
• flow matching
• repulsive distance loss
• safety-critical driving