Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures

2024-12-09 · Adrien Bolland, Gaspard Lambrechts, Damien Ernst

We introduce a new maximum entropy reinforcement learning framework based on the distribution of states and actions visited by a policy. More precisely, an intrinsic reward function is added to the reward function of the Markov decision process that shall be controlled. For each state and action, this intrinsic reward is the relative entropy of the discounted distribution of states and actions (or features from these states and actions) visited during the next time steps. We first prove that an optimal exploration policy, which maximizes the expected discounted sum of intrinsic rewards, is also a policy that maximizes a lower bound on the state-action value function of the decision process under some assumptions. We also prove that the visitation distribution used in the intrinsic reward definition is the fixed point of a contraction operator. Following, we describe how to adapt existing algorithms to learn this fixed point and compute the intrinsic rewards to enhance exploration. A new practical off-policy maximum entropy reinforcement learning algorithm is finally introduced. Empirically, exploration policies have good state-action space coverage, and high-performing control policies are computed efficiently.

📄 PDF Abstract BibTeX arXiv:2412.06655

Code (1)

adrienBolland/future-visitation-exploration 공식 구현 pytorch

Tasks

reinforcement-learning Reinforcement Learning

Maximum-Entropy Exploration with Future State-Action Visitation Measures

2026-03-19 · Adrien Bolland, Gaspard Lambrechts, Damien Ernst arxiv

Maximum entropy reinforcement learning motivates agents to explore states and actions to maximize the entropy of some distribution, typically by providing additional intrinsic rewards proportional to that entropy functio…

Reinforcement Learning

The Importance of Non-Markovianity in Maximum State Entropy Exploration

2022-02-07 · ICML Workshop URL 2021 7 · Mirco Mutti, Riccardo De Santi, Marcello Restelli

In the maximum state entropy exploration framework, an agent interacts with a reward-free environment to learn a policy that maximizes the entropy of the expected state visitations it is inducing. Hazan et al. (2019) not…

Generalized Maximum Entropy Reinforcement Learning via Reward Shaping

2021-09-29 · Feng Tao, Yongcan Cao

Entropy regularization is a commonly used technique in reinforcement learning to improve exploration and cultivate a better pre-trained policy for later adaptation. Recent studies further show that the use of entropy reg…

MuJoCo reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning

2019-09-07 · Wenjie Shi, Shiji Song, Cheng Wu

Maximum entropy deep reinforcement learning (RL) methods have been demonstrated on a range of challenging continuous tasks. However, existing methods either suffer from severe instability when training on large off-polic…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Entropy Maximization for Markov Decision Processes Under Temporal Logic Constraints

2018-07-09 · Yagiz Savas, Melkior Ornik, Murat Cubuktepe, Mustafa O. Karabag 외

We study the problem of synthesizing a policy that maximizes the entropy of a Markov decision process (MDP) subject to a temporal logic constraint. Such a policy minimizes the predictability of the paths it generates, or…

Motion Planning