Learning Policy from a Single Trajectory in Average-Reward Markov Decision Process

2026-06-15

AI Summary

This work addresses the finite-sample analysis of average-reward Markov decision processes (MDPs) under weak connectivity, a setting where existing results typically rely on strong assumptions such as ergodicity or access to a generative model and lack theoretical guarantees from a single trajectory. The paper proposes the first model-free algorithm that requires no prior knowledge of problem-specific parameters and learns a near-optimal policy using only a single sample path. By integrating value-based and policy-based approaches and rigorously characterizing the dynamics of single-trajectory learning, the method achieves sample complexities of Õ(1/ε²) and Õ(1/ε⁴) under appropriate conditions, thereby establishing the first finite-sample convergence guarantees for this challenging setting.

Abstract

While there is an extensive body of work characterizing the sample complexity of discounted cumulative-reward MDPs, finite sample analyses for average-reward MDPs have been limited, and most existing works rely on restrictive assumptions such as ergodicity or access to a generative model. In this work, we establish the first finite sample complexity guarantees from a single trajectory for weakly communicating average-reward MDPs. To this end, we study the dynamics of a single trajectory in weakly communicating MDPs and based on this analysis, we develop novel model-free methods. Notably, our value-based and policy-based methods provide finite sample complexity guarantees of $\widetilde{O}(1/\varepsilon^2)$ and $\widetilde{O}(1/\varepsilon^4)$ from a single trajectory in weakly communicating MDPs, respectively. Furthermore, we introduce the first model-free method that requires no prior knowledge of problem-dependent quantities for communicating MDPs.

Problem

• average-reward MDP
• single trajectory
• sample complexity
• weakly communicating
• policy learning

Innovation

• average-reward MDP
• single trajectory
• finite sample complexity
• model-free algorithm
• weakly communicating