Reward-Preserving Attacks For Robust Reinforcement Learning

2026-01-12

arXiv.org

AI Summary

This work addresses the challenge of balancing attack strength in adversarial reinforcement learning: overly strong perturbations degrade nominal performance, while excessively weak ones fail to enhance robustness, and the optimal perturbation intensity varies dynamically across states. To this end, the paper proposes the α-reward-preserving attack, the first method to introduce a state-adaptive perturbation mechanism that enforces, in each state, a worst-case return at least an α-fraction below the nominal return through a state-dependent perturbation magnitude η. The approach calibrates attacks by combining gradient-based direction with an off-policy trained Q-function \( Q^\pi_\alpha((s,a), \eta) \) parameterized by a radius. Experiments demonstrate that the method consistently outperforms fixed- or random-radius baselines across various perturbation radii, achieving superior robustness without compromising nominal performance.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Adversarial training in reinforcement learning (RL) is challenging because perturbations cascade through trajectories and compound over time, making fixed-strength attacks either overly destructive or too conservative. We propose reward-preserving attacks, which adapt adversarial strength so that an $\alpha$ fraction of the nominal-to-worst-case return gap remains achievable at each state. In deep RL, perturbation magnitudes $\eta$ are selected dynamically, using a learned critic $Q((s,a),\eta)$ that estimates the expected return of $\alpha$-reward-preserving rollouts. For intermediate values of $\alpha$, this adaptive training yields policies that are robust across a wide range of perturbation magnitudes while preserving nominal performance, outperforming fixed-radius and uniformly sampled-radius adversarial training.

Problem

• adversarial robustness
• reinforcement learning
• reward-preserving attacks
• state-dependent perturbations
• robustness calibration

Innovation

• reward-preserving attacks
• adversarial robustness
• adaptive perturbation
• robust reinforcement learning
• off-policy critic