A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning

2026-09-11

AI Summary

本文通过统一正则化方法并导出性能差距的上界，提出一种约束优化问题来提高深度强化学习策略在对抗性输入扰动下的鲁棒性。

Abstract

Regularization-based methods have become a standard approach for training Deep Reinforcement Learning policies against adversarial input perturbations. In this paper, we unify these methods by deriving new upper bounds on the performance gap between the nominal and worst-case policies. Each upper bound is expressed as an existing regularization objective plus a KL-divergence penalty between the nominal and worst-case policies, which further explains why adding a KL penalty improves robustness in practice. Building on these bounds, we formulate robust training as a constrained optimization problem, showing that existing methods correspond to the special case of a fixed Lagrange multiplier. We instead update the multiplier jointly with the policy to automatically tune the regularization weight. Finally, we conduct extensive adversarial evaluations across several continuous control tasks to validate our theoretical analysis.

Problem

• Regularization
• Robust Reinforcement Learning
• Adversarial Input Perturbations
• KL-divergence

Innovation

• Regularization-based methods
• KL-divergence penalty
• Constrained optimization problem
• Lagrange multiplier