On Minimizing Adversarial Counterfactual Error in Adversarial RL

2024-06-07

AI Summary

Deep reinforcement learning (DRL) policies exhibit poor robustness against adversarial observation noise, fundamentally due to attack-induced partial observability of the state. Existing defenses either degrade benign performance or adopt overly conservative strategies, without explicitly modeling this partial observability. Method: We propose a unified perspective that jointly optimizes robustness and belief inference within a partial-observability modeling framework. We introduce Adversarial Counterfactual Error (ACoE) as a principled objective balancing value accuracy and robustness under attacks, and design a model-agnostic, scalable surrogate loss—C-ACoE. Our approach integrates belief-state modeling, counterfactual value estimation under adversarial perturbations, and theoretically grounded loss optimization. Results: Evaluated on MuJoCo, Atari, and Highway benchmarks, our method significantly outperforms state-of-the-art approaches, achieving substantial gains in adversarial robustness while preserving original task performance.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Adversarial Agents
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Deep Reinforcement Learning (DRL) policies are highly susceptible to adversarial noise in observations, which poses significant risks in safety-critical scenarios. The challenge inherent to adversarial perturbations is that by altering the information observed by the agent, the state becomes only partially observable. Existing approaches address this by either enforcing consistent actions across nearby states or maximizing the worst-case value within adversarially perturbed observations. However, the former suffers from performance degradation when attacks succeed, while the latter tends to be overly conservative, leading to suboptimal performance in benign settings. We hypothesize that these limitations stem from their failing to account for partial observability directly. To this end, we introduce a novel objective called Adversarial Counterfactual Error (ACoE), defined on the beliefs about the true state and balancing value optimization with robustness. To make ACoE scalable in model-free settings, we propose the theoretically-grounded surrogate objective Cumulative-ACoE (C-ACoE). Our empirical evaluations on standard benchmarks (MuJoCo, Atari, and Highway) demonstrate that our method significantly outperforms current state-of-the-art approaches for addressing adversarial RL challenges, offering a promising direction for improving robustness in DRL under adversarial conditions. Our code is available at https://github.com/romanbelaire/acoe-robust-rl.

Problem

• Address susceptibility of DRL policies to adversarial noise
• Balance value optimization with robustness in adversarial RL
• Improve robustness in DRL under adversarial conditions

Innovation

• Introduces Adversarial Counterfactual Error (ACoE)
• Proposes Cumulative-ACoE (C-ACoE) for scalability
• Balances value optimization with robustness