Primal-Only Actor Critic Algorithm for Robust Constrained Average Cost MDPs

2025-11-07

AI Summary

This paper studies safe robust policy learning in robust constrained Markov decision processes (RCMDPs) with average-cost criteria. Addressing two fundamental theoretical challenges—the absence of strong duality and the non-contractivity of the robust Bellman operator—we propose the first primal-only actor-critic algorithm that avoids any primal-dual structure: it eschews Lagrangian duality entirely, instead leveraging policy gradients and robust dynamic programming, augmented by monotonicity analysis to handle non-contractivity. Our method is the first to achieve simultaneous ε-feasibility and ε-optimality guarantees in average-cost RCMDPs. Under constraint relaxation, the sample complexity is Õ(ε⁻⁴); without relaxation, it is Õ(ε⁻⁶)—matching the best-known rates for discounted RCMDPs.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Natural Language Processing: Safety and Robustness
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

In this work, we study the problem of finding robust and safe policies in Robust Constrained Average-Cost Markov Decision Processes (RCMDPs). A key challenge in this setting is the lack of strong duality, which prevents the direct use of standard primal-dual methods for constrained RL. Additional difficulties arise from the average-cost setting, where the Robust Bellman operator is not a contraction under any norm. To address these challenges, we propose an actor-critic algorithm for Average-Cost RCMDPs. We show that our method achieves both (epsilon)-feasibility and (epsilon)-optimality, and we establish a sample complexities of ( ilde{O}left(epsilon^{-4} ight)) and ( ilde{O}left(epsilon^{-6} ight)) with and without slackness assumption, which is comparable to the discounted setting.

Problem

• Develops robust constrained average-cost MDP algorithm
• Addresses lack of strong duality in constrained RL
• Solves non-contraction Robust Bellman operator issue

Innovation

• Actor-critic algorithm for average-cost RCMDPs
• Achieves epsilon-feasibility and epsilon-optimality
• Sample complexity comparable to discounted setting