Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning
Entropy regularization is widely used in continuous-time reinforcement learning (RL) to reduce sensitivity to environmental perturbations, yet its robustness benefits lack a rigorous theoretical foundation. This paper establishes the first robustness guarantees for entropy-regularized continuous-time Markov decision processes. We show that maximizing an entropy-regularized objective yields a lower bound on a worst-case robust RL problem with joint reward and transition perturbations. We analytically characterize the induced robust sets and prove that they expand monotonically with the regularization strength, justifying the empirical observation that stronger entropy improves robustness. In contrast to prior discrete-time analyses, our results remove the intractable state-distribution entropy term and provide guarantees invariant to action frequency. Experiments on queueing network control and market making confirm our theory, showing that entropy-regularized policies outperform greedy and $ε$-greedy baselines under dynamics perturbations.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningSimilar Papers 제목 키워드 기반
Entropy annealing for policy mirror descent in continuous time and space
Entropy regularization has been widely used in policy optimization algorithms to enhance exploration and the robustness of the optimal control; however it also introduces an additional regularization bias. This work quan…
Policy Gradient MethodsPromoting Stochasticity for Expressive Policies via a Simple and Efficient Regularization Method
Many recent reinforcement learning (RL) methods learn stochastic policies with entropy regularization for exploration and robustness. However, in continuous action spaces, integrating entropy regularization with expressi…
continuous-controlContinuous Controlreinforcement-learningReinforcement Learning (RL)Marginalized State Distribution Entropy Regularization in Policy Optimization
Entropy regularization is used to get improved optimization performance in reinforcement learning tasks. A common form of regularization is to maximize policy entropy to avoid premature convergence and lead to more stoch…
continuous-controlContinuous ControlReinforcement LearningDo You Need the Entropy Reward (in Practice)?
Maximum entropy (MaxEnt) RL maximizes a combination of the original task reward and an entropy reward. It is believed that the regularization imposed by entropy, on both policy improvement and policy evaluation, together…
Regularization Matters in Policy Optimization - An Empirical Study on Continuous Control
Deep Reinforcement Learning (Deep RL) has been receiving increasingly more attention thanks to its encouraging performance on a variety of control tasks. Yet, conventional regularization techniques in training neural ne…
continuous-controlContinuous ControlDeep Reinforcement Learning