Minimax PAC Bounds for Learning in Exogenous Contextual MDPs

2026-06-23

AI Summary

This work studies sample-efficient probably approximately correct (PAC) learning for tabular discounted Markov decision processes (MDPs) under exogenous i.i.d. contexts, where contexts influence rewards and transitions but are uncontrollable. For this setting, the authors propose a novel variance-reduced algorithm that leverages both contextual information and a transition kernel sampling oracle, achieving minimax-optimal sample complexity independent of the context space size $|\mathcal{Z}|$ while only requiring samples from the context distribution. When rewards and transitions are known, the algorithm simultaneously solves policy evaluation, optimal value estimation, and policy extraction with $\widetilde{O}(1/((1-\gamma)^3 \varepsilon^2))$ sample complexity. In the fully unknown setting, the work establishes the first matching tight upper and lower bounds, thereby characterizing the fundamental sample complexity limit of the problem.

Abstract

We study PAC learning in tabular discounted Markov decision processes with exogenous i.i.d. contexts, with discount factor $γ$, finite state space $\mathcal X$, action space $\mathcal A$, and context space $\mathcal Z$. At each time step, a context is drawn independently from an unknown distribution $μ$ and revealed before the agent acts. This context may affect both rewards and transitions, while remaining uncontrolled by the agent. Depending on the regime, the learner has access either to a sampling oracle for $μ$, to a sampling oracle for the transition kernel conditioned on state-context-action tuples, or to both. Oracles can be accessed before and during policy execution. The sample complexity is measured by a couple $(n,m)$, where $n$ is the number of calls to the sampling oracles before execution and $m$ is the number of calls to the sampling oracles during execution. When rewards and transitions are known and only the context distribution $μ$ is sampled, we give a variance-reduced algorithm that solves policy evaluation (PE), best-value estimation (BVE), and best-policy extraction (BPE) with $\left(\widetilde O\left(1/((1-γ)^3\varepsilon^2)\right), 0 \right) $ sample complexity. The rate is independent of $|\mathcal Z|$ and minimax optimal up to logarithmic factors. As a corollary, we also obtain tight rates in the case of one-step perfect look-ahead, improving upon the existing guarantees. In the fully unknown regime, where both $μ$ and P must be learned, we show that PE remains $|\mathcal Z|$-free, with matching upper and lower bounds $\bigl(\widetilde O(|\mathcal X|/((1-γ)^3\varepsilon^2)),\, \widetilde O(1/((1-γ)^2\varepsilon^2))\bigr)$.

Problem

• Exogenous Contextual MDPs
• PAC Learning
• Sample Complexity
• Minimax Bounds
• Discounted MDPs

Innovation

• exogenous contextual MDPs
• PAC learning
• variance-reduced algorithm
• minimax optimality
• sample complexity