Convergent Reinforcement Learning Algorithms for Stochastic Shortest Path Problem

2025-08-19

AI Summary

This work addresses the lack of rigorous convergence guarantees for reinforcement learning (RL) algorithms in stochastic shortest path (SSP) problems. We propose two tabular RL algorithms and one function approximation–based RL algorithm. Methodologically, we integrate dynamic programming principles with stochastic approximation theory, designing value-iteration–style update rules coupled with a step-size sequence satisfying the Robbins–Monro conditions—thereby achieving, for the first time in the SSP setting, asymptotic almost-sure convergence under multiple cost criteria (e.g., total, discounted, and average cost). Theoretically, we establish the first unified convergence framework for SSP that applies to both tabular and function approximation settings. Empirically, our algorithms demonstrate improved convergence stability and superior performance across multiple benchmark tasks; notably, the function approximation variant exhibits significantly enhanced generalization capability.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Reinforcement Learning
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In this paper we propose two algorithms in the tabular setting and an algorithm for the function approximation setting for the Stochastic Shortest Path (SSP) problem. SSP problems form an important class of problems in Reinforcement Learning (RL), as other types of cost-criteria in RL can be formulated in the setting of SSP. We show asymptotic almost-sure convergence for all our algorithms. We observe superior performance of our tabular algorithms compared to other well-known convergent RL algorithms. We further observe reliable performance of our function approximation algorithm compared to other algorithms in the function approximation setting.

Problem

• Developing convergent algorithms for Stochastic Shortest Path problems
• Addressing both tabular and function approximation RL settings
• Ensuring asymptotic almost-sure convergence in SSP solutions

Innovation

• Convergent RL algorithms for SSP
• Asymptotic almost-sure convergence guarantee
• Superior tabular and reliable approximation performance