Robust Policy Optimization via Adversarial Importance Sampling

2026-09-11

AI Summary

本文通过引入对抗重要性采样方法（Advis）来解决深度强化学习政策对输入扰动的鲁棒性问题，无需额外环境交互或辅助网络，并提供了一个PyTorch库支持快速原型设计与评估。

Abstract

Significant progress has been made in safeguarding deep reinforcement learning (DRL) policies against input perturbations. Developing robust DRL involves three main stages: algorithm design, implementation, and evaluation. In this work, we identify and address a key limitation at each stage. First, we introduce Adversarial Importance Sampling (Advis), a method that uses importance sampling over trajectories from standard training to estimate and optimize verifiable worst-case returns. Advis satisfies three desirable criteria not jointly achieved by prior work: it requires no additional environment interactions, no auxiliary networks, and captures long-term robustness. Second, we introduce advrl, a modular PyTorch library that provides clean, single-file implementations of existing robustness methods and adversarial attacks, facilitating rapid prototyping and enabling reproducible and traceable evaluations. Third, we revisit evaluation under learned adversaries and show that optimal adversarial hyperparameters do not transfer across agents, which can lead to an overestimation of robustness when using a limited set of attacker configurations. Accordingly, we evaluate policies against a large and diverse set of attackers, using 6-14x more configurations than prior work. Finally, we evaluate our approach on continuous control environments, demonstrating its effectiveness relative to existing baselines. The code is available at: https://github.com/AmineAndam04/advrl

Problem

• Deep Reinforcement Learning
• Robustness
• Adversarial Attacks

Innovation

• Adversarial Importance Sampling
• long-term robustness
• modular PyTorch library
• diverse attacker configurations