MHER: Model-based Hindsight Experience Replay

2021-07-01 · Rui Yang, Meng Fang, Lei Han, Yali Du, Feng Luo, Xiu Li

Solving multi-goal reinforcement learning (RL) problems with sparse rewards is generally challenging. Existing approaches have utilized goal relabeling on collected experiences to alleviate issues raised from sparse rewards. However, these methods are still limited in efficiency and cannot make full use of experiences. In this paper, we propose Model-based Hindsight Experience Replay (MHER), which exploits experiences more efficiently by leveraging environmental dynamics to generate virtual achieved goals. Replacing original goals with virtual goals generated from interaction with a trained dynamics model leads to a novel relabeling method, model-based relabeling (MBR). Based on MBR, MHER performs both reinforcement learning and supervised learning for efficient policy improvement. Theoretically, we also prove the supervised part in MHER, i.e., goal-conditioned supervised learning with MBR data, optimizes a lower bound on the multi-goal RL objective. Experimental results in several point-based tasks and simulated robotics environments show that MHER achieves significantly higher sample efficiency than previous model-free and model-based multi-goal methods.

📄 PDF Abstract BibTeX arXiv:2107.00306

Code (0)

등록된 구현이 없습니다.

Tasks

modelMulti-Goal Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning

2021-02-25 · Rui Yang, Jiafei Lyu, Yu Yang, Jiangpeng Yan 외

Multi-goal reinforcement learning is widely applied in planning and robot manipulation. Two main challenges in multi-goal reinforcement learning are sparse rewards and sample inefficiency. Hindsight Experience Replay (HE…

Multi-Goal Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Hindsight Curriculum Generation Based Multi-Goal Experience Replay

2021-01-01 · Xiaoyun Feng

In multi-goal tasks with sparse rewards, it is challenging to learn from tons of experiences with zero rewards. Hindsight experience replay (HER), which replays past experiences with additional heuristic goals, has shown…

Reinforcement Learning (RL)

ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay

2018-09-06 · Sameera Lanka, Tianfu Wu

Experience replay is an important technique for addressing sample-inefficiency in deep reinforcement learning (RL), but faces difficulty in learning from binary and sparse rewards due to disproportionately few successful…

continuous-control Continuous Control Deep Reinforcement Learning Reinforcement Learning+1

Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment

2021-10-28 · Tung M. Luu, Chang D. Yoo

This paper proposes a method for prioritizing the replay experience referred to as Hindsight Goal Ranking (HGR) in overcoming the limitation of Hindsight Experience Replay (HER) that generates hindsight goals based on un…

Improvements on Hindsight Learning

2018-09-16 · Ameet Deshpande, Srikanth Sarma, Ashutosh Jha, Balaraman Ravindran

Sparse reward problems are one of the biggest challenges in Reinforcement Learning. Goal-directed tasks are one such sparse reward problems where a reward signal is received only when the goal is reached. One promising w…

Policy Gradient Methods reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)