Reinforcement Learning with Intrinsically Motivated Feedback Graph for Lost-sales Inventory Control

2024-06-26

arXiv.org

AI Summary

In stockout-driven inventory control, reinforcement learning (RL) suffers from low sample efficiency and unobserved true demand due to censored sales data. To address these bottlenecks, this paper proposes a novel RL framework grounded in feedback graph modeling. We introduce, for the first time, a structured feedback graph specifically designed for stockout settings and theoretically prove that it reduces sample complexity. Furthermore, we incorporate an intrinsic motivation reward mechanism to actively guide the agent toward high-information-gain state-action pairs. Our approach integrates RL, causal feedback modeling, and theory-driven sample-efficiency analysis, achieving strong policy performance while drastically reducing required interaction data. Experiments demonstrate a 40–65% reduction in data requirements to attain equivalent performance compared to baseline methods. The implementation is publicly available.

Technology Category

• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reinforcement learning (RL) has proven to be well-performed and general-purpose in the inventory control (IC). However, further improvement of RL algorithms in the IC domain is impeded due to two limitations of online experience. First, online experience is expensive to acquire in real-world applications. With the low sample efficiency nature of RL algorithms, it would take extensive time to train the RL policy to convergence. Second, online experience may not reflect the true demand due to the lost sales phenomenon typical in IC, which makes the learning process more challenging. To address the above challenges, we propose a decision framework that combines reinforcement learning with feedback graph (RLFG) and intrinsically motivated exploration (IME) to boost sample efficiency. In particular, we first take advantage of the inherent properties of lost-sales IC problems and design the feedback graph (FG) specially for lost-sales IC problems to generate abundant side experiences aid RL updates. Then we conduct a rigorous theoretical analysis of how the designed FG reduces the sample complexity of RL methods. Based on the theoretical insights, we design an intrinsic reward to direct the RL agent to explore to the state-action space with more side experiences, further exploiting FG's power. Experimental results demonstrate that our method greatly improves the sample efficiency of applying RL in IC. Our code is available at https://anonymous.4open.science/r/RLIMFG4IC-811D/

Problem

• Improve sample efficiency in inventory control
• Address lost sales in demand data
• Enhance RL with feedback graph exploration

Innovation

• Reinforcement learning with feedback graph
• Intrinsically motivated exploration
• Improved sample efficiency in inventory control