Sample Complexity Bounds for Stochastic Shortest Path with a Generative Model

2026-04-17

AI Summary

This work investigates the sample complexity lower bounds and learnability conditions for learning ε-optimal policies in stochastic shortest path (SSP) problems with a generative model. By constructing worst-case instances and leveraging information-theoretic lower bounds combined with probabilistic analysis, it establishes that SSP becomes fundamentally unlearnable when the minimum cost \( c_{\min} = 0 \), highlighting its strictly greater difficulty compared to finite-horizon and discounted MDPs. Under a bounded hitting-time assumption, the paper proposes a new algorithm and proves a sample complexity lower bound of \( \Omega(SAB_\star^3 / (c_{\min} \varepsilon^2)) \) for both the general case and the challenging \( c_{\min} = 0 \) regime. The proposed algorithm matches this lower bound up to logarithmic factors in both settings.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We study the sample complexity of learning an $ε$-optimal policy in the Stochastic Shortest Path (SSP) problem. We first derive sample complexity bounds when the learner has access to a generative model. We show that there exists a worst-case SSP instance with $S$ states, $A$ actions, minimum cost $c_{\min}$, and maximum expected cost of the optimal policy over all states $B_{\star}$, where any algorithm requires at least $Ω(SAB_{\star}^3/(c_{\min}ε^2))$ samples to return an $ε$-optimal policy with high probability. Surprisingly, this implies that whenever $c_{\min} = 0$ an SSP problem may not be learnable, thus revealing that learning in SSPs is strictly harder than in the finite-horizon and discounted settings. We complement this lower bound with an algorithm that matches it, up to logarithmic factors, in the general case, and an algorithm that matches it up to logarithmic factors even when $c_{\min} = 0$, but only under the condition that the optimal policy has a bounded hitting time to the goal state.

Problem

• Stochastic Shortest Path
• sample complexity
• generative model
• ε-optimal policy
• learnability

Innovation

• Stochastic Shortest Path
• sample complexity
• generative model
• lower bound
• hitting time