Structured Reinforcement Learning for Combinatorial Decision-Making

2025-05-25

AI Summary

Standard reinforcement learning (RL) suffers from poor generalization and scalability in combinatorial decision-making problems—such as routing and scheduling—due to exponentially large, structured action spaces. Method: This paper proposes Structured Reinforcement Learning (SRL), a novel framework that integrates a differentiable combinatorial optimization layer into the Actor network, enabling end-to-end joint training with the Actor-Critic architecture for the first time. It employs the Fenchel–Young loss to ensure policy differentiability and probabilistic consistency, and provides a primal-dual geometric interpretation via the moment polytope’s dual space. Results: Evaluated across six dynamic environments featuring both exogenous and endogenous uncertainties, SRL achieves up to 92% higher task performance than conventional RL and imitation learning baselines, while demonstrating faster convergence and significantly improved training stability.

Technology Category

• Planning, Routing, and Scheduling: Planning/Scheduling and Learning
• Machine Learning: Structured Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Reinforcement learning (RL) is increasingly applied to real-world problems involving complex and structured decisions, such as routing, scheduling, and assortment planning. These settings challenge standard RL algorithms, which struggle to scale, generalize, and exploit structure in the presence of combinatorial action spaces. We propose Structured Reinforcement Learning (SRL), a novel actor-critic framework that embeds combinatorial optimization layers into the actor neural network. We enable end-to-end learning of the actor via Fenchel-Young losses and provide a geometric interpretation of SRL as a primal-dual algorithm in the dual of the moment polytope. Across six environments with exogenous and endogenous uncertainty, SRL matches or surpasses the performance of unstructured RL and imitation learning on static tasks and improves over these baselines by up to 92% on dynamic problems, with improved stability and convergence speed.

Problem

• Addresses scalability in RL for combinatorial action spaces
• Enhances generalization in structured decision-making tasks
• Improves stability and convergence in dynamic RL problems

Innovation

• Embeds combinatorial optimization layers into actor network
• Uses Fenchel-Young losses for end-to-end learning
• Interprets SRL as primal-dual algorithm geometrically