Soft Hindsight Experience Replay

2020-02-06 · Qiwei He, Liansheng Zhuang, Houqiang Li

Efficient learning in the environment with sparse rewards is one of the most important challenges in Deep Reinforcement Learning (DRL). In continuous DRL environments such as robotic arms control, Hindsight Experience Replay (HER) has been shown an effective solution. However, due to the brittleness of deterministic methods, HER and its variants typically suffer from a major challenge for stability and convergence, which significantly affects the final performance. This challenge severely limits the applicability of such methods to complex real-world domains. To tackle this challenge, in this paper, we propose Soft Hindsight Experience Replay (SHER), a novel approach based on HER and Maximum Entropy Reinforcement Learning (MERL), combining the failed experiences reuse and maximum entropy probabilistic inference model. We evaluate SHER on Open AI Robotic manipulation tasks with sparse rewards. Experimental results show that, in contrast to HER and its variants, our proposed SHER achieves state-of-the-art performance, especially in the difficult HandManipulation tasks. Furthermore, our SHER method is more stable, achieving very similar performance across different random seeds.

📄 PDF Abstract BibTeX arXiv:2002.02089

Code (2)

fslight1996/SHER 공식 구현 tf

YangRui2015/Modular_HER tf

Tasks

Deep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Hindsight Curriculum Generation Based Multi-Goal Experience Replay

2021-01-01 · Xiaoyun Feng

In multi-goal tasks with sparse rewards, it is challenging to learn from tons of experiences with zero rewards. Hindsight experience replay (HER), which replays past experiences with additional heuristic goals, has shown…

Reinforcement Learning (RL)

ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay

2018-09-06 · Sameera Lanka, Tianfu Wu

Experience replay is an important technique for addressing sample-inefficiency in deep reinforcement learning (RL), but faces difficulty in learning from binary and sparse rewards due to disproportionately few successful…

continuous-control Continuous Control Deep Reinforcement Learning Reinforcement Learning+1

Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment

2021-10-28 · Tung M. Luu, Chang D. Yoo

This paper proposes a method for prioritizing the replay experience referred to as Hindsight Goal Ranking (HGR) in overcoming the limitation of Hindsight Experience Replay (HER) that generates hindsight goals based on un…

Improvements on Hindsight Learning

2018-09-16 · Ameet Deshpande, Srikanth Sarma, Ashutosh Jha, Balaraman Ravindran

Sparse reward problems are one of the biggest challenges in Reinforcement Learning. Goal-directed tasks are one such sparse reward problems where a reward signal is received only when the goal is reached. One promising w…

Policy Gradient Methods reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Curriculum-guided Hindsight Experience Replay

2019-12-01 · NeurIPS 2019 12 · Meng Fang, Tianyi Zhou, Yali Du, Lei Han 외

In off-policy deep reinforcement learning, it is usually hard to collect sufficient successful experiences with sparse rewards to learn from. Hindsight experience replay (HER) enables an agent to learn from failures by t…

Deep Reinforcement Learning Diversity Reinforcement Learning