Relating Reinforcement Learning to Dynamic Programming-Based Planning

2026-03-08

AI Summary

This work aims to bridge the gap between reinforcement learning and dynamic programming in terms of objective formulation, modeling assumptions, and optimization criteria. By developing a derandomized reinforcement learning framework, it establishes both theoretical and empirical connections to value iteration and Dijkstra’s algorithm, thereby unifying cost-minimization and reward-maximization paradigms. The core contributions include identifying the equivalence conditions between these two objective formulations, demonstrating the equivalence between single-episode termination tasks and infinite-horizon learning settings, and proposing an optimization objective centered on true cost. The approach is validated in both deterministic and stochastic environments, with precise conditions under which discounting leads to objective misalignment clearly characterized. Performance comparisons are enabled through planning-oriented evaluation metrics.

Technology Category

• Machine Learning: Reinforcement Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper bridges some of the gap between optimal planning and reinforcement learning (RL), both of which share roots in dynamic programming applied to sequential decision making or optimal control. Whereas planning typically favors deterministic models, goal termination, and cost minimization, RL tends to favor stochastic models, infinite-horizon discounting, and reward maximization in addition to learning-related parameters such as the learning rate and greediness factor. A derandomized version of RL is developed, analyzed, and implemented to yield performance comparisons with value iteration and Dijkstra's algorithm using simple planning models. Next, mathematical analysis shows: 1) conditions under which cost minimization and reward maximization are equivalent, 2) conditions for equivalence of single-shot goal termination and infinite-horizon episodic learning, and 3) conditions under which discounting causes goal achievement to fail. The paper then advocates for defining and optimizing truecost, rather than inserting arbitrary parameters to guide operations. Performance studies are then extended to the stochastic case, using planning-oriented criteria and comparing value iteration to RL with learning rates and greediness factors.

Problem

• Reinforcement Learning
• Dynamic Programming
• Optimal Planning
• Cost Minimization
• Reward Maximization

Innovation

• derandomized reinforcement learning
• dynamic programming
• true cost optimization
• equivalence analysis
• value iteration