Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learning

2025-02-24

AI Summary

Finite-sample theoretical analysis for policy evaluation in robust average-reward Markov decision processes (MDPs) has long been lacking. Method: This paper establishes the first rigorous sample-complexity characterization by proving that the robust Bellman operator is a contraction under the span semi-norm, and then designing a geometrically truncated multilevel Monte Carlo (MLMC) estimator based on this property. The estimator achieves exponential bias decay and controlled variance. Results: The proposed framework attains an order-optimal sample complexity of $ ilde{mathcal{O}}(epsilon^{-2})$. Unlike prior works offering only asymptotic convergence guarantees, this study fills a fundamental theoretical gap in finite-sample analysis for robust policy evaluation, thereby significantly enhancing algorithmic interpretability and practical applicability in robust reinforcement learning.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Intelligent Robots: State Estimation
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We present the first finite-sample analysis for policy evaluation in robust average-reward Markov Decision Processes (MDPs). Prior works in this setting have established only asymptotic convergence guarantees, leaving open the question of sample complexity. In this work, we address this gap by establishing that the robust Bellman operator is a contraction under the span semi-norm, and developing a stochastic approximation framework with controlled bias. Our approach builds upon Multi-Level Monte Carlo (MLMC) techniques to estimate the robust Bellman operator efficiently. To overcome the infinite expected sample complexity inherent in standard MLMC, we introduce a truncation mechanism based on a geometric distribution, ensuring a finite constant sample complexity while maintaining a small bias that decays exponentially with the truncation level. Our method achieves the order-optimal sample complexity of $ ilde{mathcal{O}}(epsilon^{-2})$ for robust policy evaluation and robust average reward estimation, marking a significant advancement in robust reinforcement learning theory.

Problem

• Finite-sample analysis for robust policy evaluation
• Develops stochastic approximation with controlled bias
• Achieves optimal sample complexity in robust RL

Innovation

• Robust Bellman operator contraction
• Multi-Level Monte Carlo techniques
• Geometric distribution truncation mechanism