Entropy-Regularized Reinforcement Learning for Linear-Quadratic Stackelberg Differential Games in Regime-Switching Diffusion Models

2026-06-27 · Congde Hu, Danping Li, Lin Xu, Wenying Xu arxiv

Stackelberg differential games (SDGs) provide a powerful framework for hierarchical decision-making in stochastic and continuous-time environments, yet their solution remains computationally challenging due to the complexity of traditional dynamic programming and Hamilton-Jacobi-Bellman-Isaacs (HJBI) methods, especially in high-dimensional systems. This paper proposes an entropy-regularized reinforcement learning (ERRL) approach for linear-quadratic SDGs (LQ-SDGs) within a continuous-time diffusion framework governed by Markovian regime switching. The key innovation lies in deriving exploratory weakly-coupled HJBI equations with entropy regularization, which promotes stochastic policies that actively avoid suboptimal equilibria -- a limitation of classical SDG methods. Neural networks are integrated to approximate regime-dependent value functions and solve high-dimensional partial differential equations (PDEs) efficiently, while a novel sampling technique enhances computational tractability. Numerical results demonstrate the effectiveness of the framework compared to conventional approaches, particularly in escaping suboptimal traps through exploratory policies. The study highlights the critical role of entropy regularization and neural network approximations in achieving robust solutions for hierarchical decision-making problems under abrupt environmental shifts.

📄 PDF Abstract BibTeX arXiv:2606.28671

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Global Convergence of Policy Gradient for Entropy Regularized Linear-Quadratic Control with Multiplicative Noise

2025-10-03 · Gabriel Diaz, Lucky Li, Wenhao Zhang arxiv

Reinforcement Learning (RL) has emerged as a powerful framework for sequential decision-making in dynamic environments, particularly when system parameters are unknown. This paper investigates RL-based control for entrop…

Reinforcement Learning

Choquet regularization for reinforcement learning

2022-08-17 · Xia Han, Ruodu Wang, Xun Yu Zhou

We propose \emph{Choquet regularizers} to measure and manage the level of exploration for reinforcement learning (RL), and reformulate the continuous-time entropy-regularized RL problem of Wang et al. (2020, JMLR, 21(198…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Fast Policy Learning for Linear Quadratic Control with Entropy Regularization

2023-11-23 · Xin Guo, Xinyu Li, Renyuan Xu

This paper proposes and analyzes two new policy learning methods: regularized policy gradient (RPG) and iterative policy optimization (IPO), for a class of discounted linear-quadratic control (LQC) problems over an infin…

Actions Speak What You Want: Provably Sample-Efficient Reinforcement Learning of the Quantal Stackelberg Equilibrium from Strategic Feedbacks

2023-07-26 · Siyu Chen, Mengdi Wang, Zhuoran Yang

We study reinforcement learning (RL) for learning a Quantal Stackelberg Equilibrium (QSE) in an episodic Markov game with a leader-follower structure. In specific, at the outset of the game, the leader announces her poli…

Decision Making LEMMA Reinforcement Learning (RL)

Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization

2020-07-13 · Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei 외

Natural policy gradient (NPG) methods are among the most widely used policy optimization algorithms in contemporary reinforcement learning. This class of methods is often applied in conjunction with entropy regularizatio…

Policy Gradient Methods