Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning

2026-07-03

AI Summary

This work addresses the sensitivity of policies to environmental perturbations and the lack of theoretical robustness guarantees for entropy regularization in continuous-time reinforcement learning. For the first time, it establishes a formal connection between entropy regularization and worst-case robust reinforcement learning within continuous-time Markov decision processes, proving their equivalence to a robust optimization problem that simultaneously accounts for perturbations in both rewards and state transitions. The induced uncertainty set is explicitly characterized, shown to expand monotonically with the strength of entropy regularization, and crucially independent of action frequency. Empirical evaluations on queueing network control and market-making tasks demonstrate that entropy-regularized policies significantly outperform greedy and ε-greedy baselines under dynamic perturbations.

Abstract

Entropy regularization is widely used in continuous-time reinforcement learning (RL) to reduce sensitivity to environmental perturbations, yet its robustness benefits lack a rigorous theoretical foundation. This paper establishes the first robustness guarantees for entropy-regularized continuous-time Markov decision processes. We show that maximizing an entropy-regularized objective yields a lower bound on a worst-case robust RL problem with joint reward and transition perturbations. We analytically characterize the induced robust sets and prove that they expand monotonically with the regularization strength, justifying the empirical observation that stronger entropy improves robustness. In contrast to prior discrete-time analyses, our results remove the intractable state-distribution entropy term and provide guarantees invariant to action frequency. Experiments on queueing network control and market making confirm our theory, showing that entropy-regularized policies outperform greedy and $ε$-greedy baselines under dynamics perturbations.

Problem

• entropy regularization
• robustness
• continuous-time reinforcement learning
• Markov decision processes
• policy robustness

Innovation

• entropy regularization
• continuous-time reinforcement learning
• robustness guarantees
• Markov decision processes
• policy robustness