DARLING: Detection Augmented Reinforcement Learning with Non-Stationary Guarantees

2026-04-17

AI Summary

This work addresses the challenge of reinforcement learning in piecewise-stationary environments where the timing and nature of non-stationary changes are unknown. Existing model-free algorithms lack both theoretical guarantees and computational efficiency in such settings. To bridge this gap, we propose DARLING—a modular framework that integrates change-point detection into model-free reinforcement learning, applicable to both tabular and linear Markov decision processes without requiring prior knowledge of change times or non-stationarity structure. We establish the first minimax lower bound on dynamic regret under this setting and develop the first algorithm that nearly matches this fundamental limit. Theoretical analysis demonstrates that our method achieves superior dynamic regret compared to existing approaches, and experiments on standard benchmarks consistently outperform state-of-the-art algorithms, highlighting its strong adaptability and robustness.

Technology Category

• Machine Learning: Reinforcement Learning
• Reasoning under Uncertainty: Sequential Decision Making
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

We study model-free reinforcement learning (RL) in non-stationary finite-horizon episodic Markov decision processes (MDPs) without prior knowledge of the non-stationarity. We focus on the piecewise-stationary (PS) setting, where both the reward and transition dynamics can change an arbitrary number of times. We propose Detection Augmented Reinforcement Learning (DARLING), a modular wrapper for PS-RL that applies to both tabular and linear MDPs, without knowledge of the changes. Under certain change-point separation and reachability conditions, DARLING improves the best available dynamic regret bounds in both settings and yields strong empirical performance. We further establish the first minimax lower bounds for PS-RL in tabular and linear MDPs, showing that DARLING is the first nearly optimal algorithm. Experiments on standard benchmarks demonstrate that DARLING consistently surpasses the state-of-the-art methods across diverse non-stationary scenarios.

Problem

• non-stationary reinforcement learning
• piecewise-stationary MDPs
• change-point detection
• dynamic regret
• model-free RL

Innovation

• non-stationary reinforcement learning
• piecewise-stationary MDPs
• change-point detection
• dynamic regret
• minimax optimality