Efficient Algorithms for Robust Markov Decision Processes with $s$-Rectangular Ambiguity Sets

2026-02-05

AI Summary

This work addresses the degradation in out-of-sample performance of traditional Markov decision processes (MDPs) under model uncertainty, which arises from their reliance on exact transition kernels. To overcome this limitation, the paper introduces the first unified and efficient solution framework applicable to a broad class of $s$-rectangular ambiguity sets—including those based on the 1-norm, 2-norm, and $\phi$-divergences. The approach computes robust policies by independently optimizing worst-case transition probabilities at each state, integrating dynamic programming with tailored convex optimization techniques. Empirical results demonstrate that the proposed algorithm achieves speedups of several orders of magnitude over existing commercial solvers on both synthetic and standard benchmark instances, while incurring only a logarithmic-factor overhead compared to solving classical MDPs.

Technology Category

• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Sustainability of Web economics

Abstract

Robust Markov decision processes (MDPs) have attracted significant interest due to their ability to protect MDPs from poor out-of-sample performance in the presence of ambiguity. In contrast to classical MDPs, which account for stochasticity by modeling the dynamics through a stochastic process with a known transition kernel, a robust MDP additionally accounts for ambiguity by optimizing against the most adverse transition kernel from an ambiguity set constructed via historical data. In this paper, we develop a unified solution framework for a broad class of robust MDPs with $s$-rectangular ambiguity sets, where the most adverse transition probabilities are considered independently for each state. Using our algorithms, we show that $s$-rectangular robust MDPs with $1$- and $2$-norm as well as $\phi$-divergence ambiguity sets can be solved several orders of magnitude faster than with state-of-the-art commercial solvers, and often only a logarithmic factor slower than classical MDPs. We demonstrate the favorable scaling properties of our algorithms on a range of synthetically generated as well as standard benchmark instances.

Problem

• Robust Markov Decision Processes
• s-rectangular ambiguity sets
• model ambiguity
• out-of-sample performance

Innovation

• Robust MDPs
• s-rectangular ambiguity sets
• efficient algorithms
• φ-divergence
• norm-based ambiguity