Sparse Threats, Focused Defense: Criticality-Aware Robust Reinforcement Learning for Safe Autonomous Driving

2026-01-05

arXiv.org

AI Summary

This work addresses the sensitivity of existing reinforcement learning methods to perturbations in autonomous driving and the inadequacy of conventional adversarial training, which overlooks the sparsity of safety-critical events and the asymmetry between attack and defense. To enhance robustness in deployment, the authors propose Criticality-Aware Robust Reinforcement Learning (CARRL), which uniquely models the attacker–defender interaction as a general-sum game. CARRL introduces a collaborative framework comprising a Risk-Exposing Adversary (REA) and a Risk-Targeted Robust Agent (RTRA), augmented with dual replay buffers, policy consistency constraints, and a risk-focusing mechanism to effectively handle the scarcity of both safety-critical events and adversarial data. Experimental results demonstrate that CARRL reduces collision rates by at least 22.66% across diverse scenarios compared to state-of-the-art baselines, significantly improving both safety and driving efficiency.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Adversarial Agents
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Reinforcement learning (RL) has shown considerable potential in autonomous driving (AD), yet its vulnerability to perturbations remains a critical barrier to real-world deployment. As a primary countermeasure, adversarial training improves policy robustness by training the AD agent in the presence of an adversary that deliberately introduces perturbations. Existing approaches typically model the interaction as a zero-sum game with continuous attacks. However, such designs overlook the inherent asymmetry between the agent and the adversary and then fail to reflect the sparsity of safety-critical risks, rendering the achieved robustness inadequate for practical AD scenarios. To address these limitations, we introduce criticality-aware robust RL (CARRL), a novel adversarial training approach for handling sparse, safety-critical risks in autonomous driving. CARRL consists of two interacting components: a risk exposure adversary (REA) and a risk-targeted robust agent (RTRA). We model the interaction between the REA and RTRA as a general-sum game, allowing the REA to focus on exposing safety-critical failures (e.g., collisions) while the RTRA learns to balance safety with driving efficiency. The REA employs a decoupled optimization mechanism to better identify and exploit sparse safety-critical moments under a constrained budget. However, such focused attacks inevitably result in a scarcity of adversarial data. The RTRA copes with this scarcity by jointly leveraging benign and adversarial experiences via a dual replay buffer and enforces policy consistency under perturbations to stabilize behavior. Experimental results demonstrate that our approach reduces the collision rate by at least 22.66\% across all cases compared to state-of-the-art baseline methods.

Problem

• sparse threats
• safety-critical risks
• robust reinforcement learning
• autonomous driving
• adversarial training

Innovation

• criticality-aware
• sparse threats
• general-sum game
• adversarial training
• dual replay buffer