Near-Constant Strong Violation and Last-Iterate Convergence for Online CMDPs via Decaying Safety Margins

2026-02-11

AI Summary

This work addresses the challenge in online constrained Markov decision processes (CMDPs) where existing methods struggle to simultaneously achieve sublinear regret in cumulative reward and bounded constraint violation. The authors propose FlexDOME, an algorithm that integrates time-varying safety margins and margin-based regularization within a primal-dual framework. FlexDOME is the first to guarantee last-iterate convergence in a non-asymptotic sense while attaining near-constant cumulative constraint violation ($\tilde{O}(1)$) and sublinear reward regret. The theoretical analysis leverages a policy-dual Lyapunov function combined with asymptotic dominance techniques to establish these guarantees. Empirical evaluations further demonstrate the algorithm’s effectiveness in practice.

Technology Category

• Constraint Satisfaction and Optimization: Constraint Optimization
• Search and Optimization: Mixed Discrete/Continuous Search
• Multiagent Systems: Mechanism Design

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

We study safe online reinforcement learning in Constrained Markov Decision Processes (CMDPs) under strong regret and violation metrics, which forbid error cancellation over time. Existing primal-dual methods that achieve sublinear strong reward regret inevitably incur growing strong constraint violation or are restricted to average-iterate convergence due to inherent oscillations. To address these limitations, we propose the Flexible safety Domain Optimization via Margin-regularized Exploration (FlexDOME) algorithm, the first to provably achieve near-constant $\tilde{O}(1)$ strong constraint violation alongside sublinear strong regret and non-asymptotic last-iterate convergence. FlexDOME incorporates time-varying safety margins and regularization terms into the primal-dual framework. Our theoretical analysis relies on a novel term-wise asymptotic dominance strategy, where the safety margin is rigorously scheduled to asymptotically majorize the functional decay rates of the optimization and statistical errors, thereby clamping cumulative violations to a near-constant level. Furthermore, we establish non-asymptotic last-iterate convergence guarantees via a policy-dual Lyapunov argument. Experiments corroborate our theoretical findings.

Problem

• Constrained Markov Decision Processes
• online reinforcement learning
• strong regret
• strong constraint violation
• last-iterate convergence

Innovation

• decaying safety margins
• strong constraint violation
• last-iterate convergence
• primal-dual reinforcement learning
• non-asymptotic convergence