Near-Optimal Sample Complexities of Divergence-based S-rectangular Distributionally Robust Reinforcement Learning

2025-05-18

AI Summary

This paper addresses the poor policy generalization in distributionally robust reinforcement learning (DR-RL) arising from distributional shifts between training and test environments, focusing on *f*-divergence-based adversarial modeling under the *S*-rectangular uncertainty set. We propose a novel algorithm integrating empirical iteration with concentration analysis. To our knowledge, this is the first method achieving a near-optimal sample complexity bound of $widetilde{O}(|S||A|(1-gamma)^{-4}varepsilon^{-2})$ under the *S*-rectangular setting—simultaneously tight in state space size $|S|$, action space size $|A|$, and accuracy $varepsilon$. This result breaks the theoretical bottleneck imposed by prior *SA*-rectangular analyses. We empirically validate rapid convergence on inventory control and worst-case task benchmarks. Our work establishes the first sample-efficiency guarantee for DR-RL that is both theoretically optimal and practically implementable.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Game Theory and Economic Paradigms: Adversarial Learning
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Distributionally robust reinforcement learning (DR-RL) has recently gained significant attention as a principled approach that addresses discrepancies between training and testing environments. To balance robustness, conservatism, and computational traceability, the literature has introduced DR-RL models with SA-rectangular and S-rectangular adversaries. While most existing statistical analyses focus on SA-rectangular models, owing to their algorithmic simplicity and the optimality of deterministic policies, S-rectangular models more accurately capture distributional discrepancies in many real-world applications and often yield more effective robust randomized policies. In this paper, we study the empirical value iteration algorithm for divergence-based S-rectangular DR-RL and establish near-optimal sample complexity bounds of $widetilde{O}(|mathcal{S}||mathcal{A}|(1-gamma)^{-4}varepsilon^{-2})$, where $varepsilon$ is the target accuracy, $|mathcal{S}|$ and $|mathcal{A}|$ denote the cardinalities of the state and action spaces, and $gamma$ is the discount factor. To the best of our knowledge, these are the first sample complexity results for divergence-based S-rectangular models that achieve optimal dependence on $|mathcal{S}|$, $|mathcal{A}|$, and $varepsilon$ simultaneously. We further validate this theoretical dependence through numerical experiments on a robust inventory control problem and a theoretical worst-case example, demonstrating the fast learning performance of our proposed algorithm.

Problem

• Analyzes sample complexity in S-rectangular DR-RL models
• Addresses training-testing environment discrepancies robustly
• Establishes near-optimal bounds for divergence-based algorithms

Innovation

• Uses divergence-based S-rectangular DR-RL model
• Empirical value iteration algorithm for optimization
• Achieves near-optimal sample complexity bounds