Maximum Entropy-Regularized Multi-Goal Reinforcement Learning

2019-05-21 · Rui Zhao, Xudong Sun, Volker Tresp

In Multi-Goal Reinforcement Learning, an agent learns to achieve multiple goals with a goal-conditioned policy. During learning, the agent first collects the trajectories into a replay buffer, and later these trajectories are selected randomly for replay. However, the achieved goals in the replay buffer are often biased towards the behavior policies. From a Bayesian perspective, when there is no prior knowledge about the target goal distribution, the agent should learn uniformly from diverse achieved goals. Therefore, we first propose a novel multi-goal RL objective based on weighted entropy. This objective encourages the agent to maximize the expected return, as well as to achieve more diverse goals. Secondly, we developed a maximum entropy-based prioritization framework to optimize the proposed objective. For evaluation of this framework, we combine it with Deep Deterministic Policy Gradient, both with or without Hindsight Experience Replay. On a set of multi-goal robotic tasks of OpenAI Gym, we compare our method with other baselines and show promising improvements in both performance and sample-efficiency.

📄 PDF Abstract BibTeX arXiv:1905.08786

Code (3)

ruizhaogit/mep 공식 구현 tf

Baichenjia/BHER tf

YangRui2015/Modular_HER tf

Tasks

Multi-Goal Reinforcement Learning OpenAI Gym reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Implicit Policy for Reinforcement Learning

2018-06-10 · Yunhao Tang, Shipra Agrawal

We introduce Implicit Policy, a general class of expressive policies that can flexibly represent complex action distributions in reinforcement learning, with efficient algorithms to compute entropy regularized policy gra…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Regularized Opponent Model with Maximum Entropy Objective

2019-05-17 · Zheng Tian, Ying Wen, Zhichen Gong, Faiz Punakkath 외

In a single-agent setting, reinforcement learning (RL) tasks can be cast into an inference problem by introducing a binary random variable o, which stands for the "optimality". In this paper, we redefine the binary rando…

model Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Regularized Inverse Reinforcement Learning

2020-10-07 · ICLR 2021 1 · Wonseok Jeon, Chen-Yang Su, Paul Barde, Thang Doan 외

Inverse Reinforcement Learning (IRL) aims to facilitate a learner's ability to imitate expert behavior by acquiring reward functions that explain the expert's decisions. Regularized IRL applies strongly convex regularize…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Sampling

2026-04-10 · Xubin Zhou, Yipeng Yang, Zhan Li arxiv

Maximum entropy reinforcement learning (MaxEnt RL) has become a standard framework for sequential decision making, yet its standard Gaussian policy parameterization is inherently unimodal, limiting its ability to model c…

Reinforcement Learning Decision Making

Fast Rates for Maximum Entropy Exploration

2023-03-14 · Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines 외

We address the challenge of exploration in reinforcement learning (RL) when the agent operates in an unknown environment with sparse or no rewards. In this work, we study the maximum entropy exploration problem of two di…

Reinforcement Learning (RL)