A Model Selection Approach for Corruption Robust Reinforcement Learning

2021-10-07

International Conference on Algorithmic Learning Theory

AI Summary

This work studies robust policy learning for reinforcement learning under adversarial corruption: specifically, in both tabular and linear Markov decision processes (MDPs), where both the transition dynamics and reward function suffer unknown total adversarial contamination (“corruption”). We propose the first algorithmic framework that achieves worst-case optimal regret *without prior knowledge* of the corruption magnitude. Our method innovatively integrates adaptive confidence set construction, hierarchical policy evaluation, and confidence-interval-based model selection. For tabular MDPs, it attains a regret bound of $widetilde{O}(min{1/Delta,sqrt{T}} + C)$; for linear MDPs, it achieves $widetilde{O}(sqrt{(1+C)T})$, where $C$ denotes the total corruption level and $T$ the horizon. These bounds significantly improve upon prior results. Moreover, our framework establishes a unified, scalable robust modeling paradigm applicable to broad classes of sequential decision-making problems under adversarial uncertainty.

Abstract

We develop a model selection approach to tackle reinforcement learning with adversarial corruption in both transition and reward. For finite-horizon tabular MDPs, without prior knowledge on the total amount of corruption, our algorithm achieves a regret bound of $widetilde{mathcal{O}}(min{frac{1}{Delta}, sqrt{T}}+C)$ where $T$ is the number of episodes, $C$ is the total amount of corruption, and $Delta$ is the reward gap between the best and the second-best policy. This is the first worst-case optimal bound achieved without knowledge of $C$, improving previous results of Lykouris et al. (2021); Chen et al. (2021); Wu et al. (2021). For finite-horizon linear MDPs, we develop a computationally efficient algorithm with a regret bound of $widetilde{mathcal{O}}(sqrt{(1+C)T})$, and another computationally inefficient one with $widetilde{mathcal{O}}(sqrt{T}+C)$, improving the result of Lykouris et al. (2021) and answering an open question by Zhang et al. (2021b). Finally, our model selection framework can be easily applied to other settings including linear bandits, linear contextual bandits, and MDPs with general function approximation, leading to several improved or new results.

Problem

• Reinforcement Learning
• Malicious Interference
• Stability in Markov Decision Processes

Innovation

• Robust Reinforcement Learning
• Linear MDP Games
• Optimization Strategies