Learning Adversarial MDPs with Stochastic Hard Constraints

2024-03-06

arXiv.org

AI Summary

This paper studies online learning for constrained Markov decision processes (CMDPs) under adversarial loss functions and stochastic hard constraints, with feedback limited to bandit-style observations. Addressing non-stationary environments, it is the first to jointly model adversarial objective losses and random hard constraints. The authors propose three novel algorithms grounded in an optimistic game-theoretic framework, leveraging Slater condition analysis, confidence-region construction, and constraint-correction mechanisms. These methods overcome limitations of prior CMDP works—namely, reliance on soft constraints or stationarity assumptions—and achieve sublinear regret across three distinct settings. Notably, the latter two algorithms guarantee constraint satisfaction with high probability at each round and incur only constant cumulative positive constraint violation, respectively. Theoretical analysis further establishes that dependence on the Slater parameter is unavoidable.

Technology Category

• Machine Learning: Online Learning & Bandits
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Economics, Online Markets and Human Computation: Social networks and social learning
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We study online learning in constrained Markov decision processes (CMDPs) with adversarial losses and stochastic hard constraints, under bandit feedback. We consider three scenarios. In the first one, we address general CMDPs, where we design an algorithm attaining sublinear regret and cumulative positive constraints violation. In the second scenario, under the mild assumption that a policy strictly satisfying the constraints exists and is known to the learner, we design an algorithm that achieves sublinear regret while ensuring that constraints are satisfied at every episode with high probability. In the last scenario, we only assume the existence of a strictly feasible policy, which is not known to the learner, and we design an algorithm attaining sublinear regret and constant cumulative positive constraints violation. Finally, we show that in the last two scenarios, a dependence on the Slater's parameter is unavoidable. To the best of our knowledge, our work is the first to study CMDPs involving both adversarial losses and hard constraints. Thus, our algorithms can deal with general non-stationary environments subject to requirements much stricter than those manageable with existing ones, enabling their adoption in a much wider range of applications.

Problem

• Online learning in constrained MDPs
• Adversarial losses with stochastic constraints
• Sublinear regret and constraint violation control

Innovation

• Adversarial losses handling
• Stochastic hard constraints
• Sublinear regret algorithms