Regularized Policies are Reward Robust

2021-01-18 · Hisham Husain, Kamil Ciosek, Ryota Tomioka

Entropic regularization of policies in Reinforcement Learning (RL) is a commonly used heuristic to ensure that the learned policy explores the state-space sufficiently before overfitting to a local optimal policy. The primary motivation for using entropy is for exploration and disambiguating optimal policies; however, the theoretical effects are not entirely understood. In this work, we study the more general regularized RL objective and using Fenchel duality; we derive the dual problem which takes the form of an adversarial reward problem. In particular, we find that the optimal policy found by a regularized objective is precisely an optimal policy of a reinforcement learning problem under a worst-case adversarial reward. Our result allows us to reinterpret the popular entropic regularization scheme as a form of robustification. Furthermore, due to the generality of our results, we apply to other existing regularization schemes. Our results thus give insights into the effects of regularization of policies and deepen our understanding of exploration through robust rewards at large.

📄 PDF Abstract BibTeX arXiv:2101.07012

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Regularized Reward-Punishment Reinforcement Learning

2026-06-26 · Jiexin Wang, Eiji Uchibe arxiv

We propose KL-Coupled Policy Regularization (KCPR), a policy coordination framework for Reward-Punishment Reinforcement Learning (RPRL). Based on KCPR, we derive KL-Coupled Soft Optimality (KCSO) and develop its deep rea…

Reinforcement Learning

A Relation Analysis of Markov Decision Process Frameworks

2020-08-18 · Tien Mai, Patrick Jaillet

We study the relation between different Markov Decision Process (MDP) frameworks in the machine learning and econometrics literatures, including the standard MDP, the entropy and general regularized MDP, and stochastic M…

Deep Reinforcement Learning Econometrics Relation

Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization

2024-10-20 · Timofei Gritsaev, Nikita Morozov, Sergey Samsonov, Daniil Tiapkin

Generative Flow Networks (GFlowNets) are a family of generative models that learn to sample objects with probabilities proportional to a given reward function. The key concept behind GFlowNets is the use of two stochasti…

Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning

2026-07-26 · Zahra Abdalla Elashaal, Afef Hfaiedh, Nahla Khraief, Issmail Ellabib 외 arxiv

Exploration in sparse-reward long-horizon tasks poses significant challenges for reinforcement learning. To address these challenges, we propose a two-level Hierarchical Reinforcement Learning (HRL) framework. The first …

Hierarchical Reinforcement Learning Continuous Control

Adversarial Imitation via Variational Inverse Reinforcement Learning

2018-09-17 · ICLR 2019 5 · Ahmed H. Qureshi, Byron Boots, Michael C. Yip

We consider a problem of learning the reward and policy from expert examples under unknown dynamics. Our proposed method builds on the framework of generative adversarial networks and introduces the empowerment-regulariz…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Transfer Learning