Heuristics for Combinatorial Optimization via Value-based Reinforcement Learning: A Unified Framework and Analysis

2025-12-09

AI Summary

Combinatorial optimization (CO) lacks rigorous reinforcement learning (RL) theory—particularly value-function-based foundations—for guaranteed performance. Method: This paper proposes the first unified value-based RL framework for CO, modeling CO problems as undiscounted Markov decision processes (MDPs) and providing easily verifiable sufficient conditions for equivalence to an MDP. Under mild assumptions, it establishes rigorous convergence of value-iteration-based RL algorithms to near-optimal solutions, quantifying the optimality gap in terms of problem size, RL estimation error, and batch size. It further uncovers intrinsic mechanisms underlying the empirical success of deep Q-learning in CO, identifying state embedding quality and projected gradient descent as critical determinants of convergence rate. Contribution/Results: This work delivers the first verifiable modeling criteria for value-based RL in CO, coupled with dual theoretical guarantees—convergence and approximation bounds—thereby bridging a fundamental gap between RL methodology and CO theory.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Optimization

Application Category

• Economics, Online Markets and Human Computation: Social networks and social learning
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Since the 1990s, considerable empirical work has been carried out to train statistical models, such as neural networks (NNs), as learned heuristics for combinatorial optimization (CO) problems. When successful, such an approach eliminates the need for experts to design heuristics per problem type. Due to their structure, many hard CO problems are amenable to treatment through reinforcement learning (RL). Indeed, we find a wealth of literature training NNs using value-based, policy gradient, or actor-critic approaches, with promising results, both in terms of empirical optimality gaps and inference runtimes. Nevertheless, there has been a paucity of theoretical work undergirding the use of RL for CO problems. To this end, we introduce a unified framework to model CO problems through Markov decision processes (MDPs) and solve them using RL techniques. We provide easy-to-test assumptions under which CO problems can be formulated as equivalent undiscounted MDPs that provide optimal solutions to the original CO problems. Moreover, we establish conditions under which value-based RL techniques converge to approximate solutions of the CO problem with a guarantee on the associated optimality gap. Our convergence analysis provides: (1) a sufficient rate of increase in batch size and projected gradient descent steps at each RL iteration; (2) the resulting optimality gap in terms of problem parameters and targeted RL accuracy; and (3) the importance of a choice of state-space embedding. Together, our analysis illuminates the success (and limitations) of the celebrated deep Q-learning algorithm in this problem context.

Problem

• Modeling combinatorial optimization as Markov decision processes for reinforcement learning.
• Establishing conditions for value-based RL convergence with optimality guarantees.
• Analyzing deep Q-learning's effectiveness and limitations in solving CO problems.

Innovation

• Unified Markov decision process framework for combinatorial optimization
• Value-based reinforcement learning with convergence guarantees
• State-space embedding analysis for deep Q-learning success