Online Markov Decision Processes with Terminal Law Constraints

2026-01-12

AI Summary

This work addresses the challenge of applying reinforcement learning to realistic online settings where traditional assumptions—such as episodic resets or infinite horizons—are invalid. The authors propose a reset-free periodic reinforcement learning framework that learns policies cycling back to the initial state distribution after a fixed number of steps, while minimizing cumulative loss. They introduce a novel performance metric termed “periodic regret” and develop an online algorithm for tabular Markov decision processes with M homogeneous agents, jointly optimizing loss and satisfying terminal distribution constraints. The algorithm provides the first non-asymptotic theoretical guarantee for multi-agent, reset-free online learning, achieving a sublinear periodic regret of Õ(T^{3/4}) and filling a critical theoretical gap for the case M > 1.

Technology Category

• Machine Learning: Online Learning & Bandits
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Traditional reinforcement learning usually assumes either episodic interactions with resets or continuous operation to minimize average or cumulative loss. While episodic settings have many theoretical results, resets are often unrealistic in practice. The infinite-horizon setting avoids this issue but lacks non-asymptotic guarantees in online scenarios with unknown dynamics. In this work, we move towards closing this gap by introducing a reset-free framework called the periodic framework, where the goal is to find periodic policies: policies that not only minimize cumulative loss but also return the agents to their initial state distribution after a fixed number of steps. We formalize the problem of finding optimal periodic policies and identify sufficient conditions under which it is well-defined for tabular Markov decision processes. To evaluate algorithms in this framework, we introduce the periodic regret, a measure that balances cumulative loss with the terminal law constraint. We then propose the first algorithms for computing periodic policies in two multi-agent settings and show they achieve sublinear periodic regret of order $\tilde O(T^{3/4})$. This provides the first non-asymptotic guarantees for reset-free learning in the setting of $M$ homogeneous agents, for $M>1$.

Problem

• Online Markov Decision Processes
• Terminal Law Constraints
• Periodic Policies
• Reset-free Learning
• Non-asymptotic Guarantees

Innovation

• Periodic Policy
• Reset-Free Reinforcement Learning
• Periodic Regret
• Markov Decision Processes
• Non-Asymptotic Guarantee