Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations

2026-09-09

AI Summary

本文针对状态扰动下的风险敏感强化学习问题，通过引入φ-散度松弛方法将问题转化为凸优化，并提出一种经验方法选择训练中的风险规避参数以提高认证下界。

Abstract

Reinforcement learning (RL) agents deployed in real-world environments are often vulnerable to adversarial perturbations in state observations, creating risks in safety-critical applications. Certification methods can improve robustness against adversarial perturbations by providing lower bounds on expected cumulative rewards. Existing certification methods, however, mainly focus on risk-neutral objectives. In this paper, we extend certification methods to risk-sensitive objectives by establishing lower bounds on the exponential utility of cumulative rewards under $l_{p}$-norm-bounded state adversarial perturbations ($1\leq p <\infty$). By introducing a $φ$-divergence relaxation of the perturbation set, we formulate the risk-sensitive certification problem as a convex optimization and derive its dual to obtain a tractable approximation of the certified lower bound. We further propose an empirical method that improves certified lower bounds by selecting the training risk-aversion parameter $β$ independently of the risk level used during evaluation. Experiments on both OpenAI Gym environments and a machine replacement problem show that, compared to risk-neutral training, risk-averse training generally yields policies with higher certified lower bounds, particularly under larger perturbation budgets. Moreover, under both risk-neutral and risk-averse evaluation settings, increasing risk aversion during training leads to non-monotonic certification performance, where certified lower bounds initially improve but eventually decrease due to overly conservative policies.

Problem

• risk-sensitive reinforcement learning
• adversarial state perturbations
• certification methods
• exponential utility
• robustness

Innovation

• risk-sensitive reinforcement learning
• certified lower bounds
• adversarial state perturbations
• $\u03c6$-divergence relaxation
• convex optimization