Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks

2021-10-05 · Robert McCarthy, Qiang Wang, Stephen J. Redmond

Model-based reinforcement learning is a promising learning strategy for practical robotic applications due to its improved data-efficiency versus model-free counterparts. However, current state-of-the-art model-based methods rely on shaped reward signals, which can be difficult to design and implement. To remedy this, we propose a simple model-based method tailored for sparse-reward multi-goal tasks that foregoes the need for complicated reward engineering. This approach, termed Imaginary Hindsight Experience Replay, minimises real-world interactions by incorporating imaginary data into policy updates. To improve exploration in the sparse-reward setting, the policy is trained with standard Hindsight Experience Replay and endowed with curiosity-based intrinsic rewards. Upon evaluation, this approach provides an order of magnitude increase in data-efficiency on average versus the state-of-the-art model-free method in the benchmark OpenAI Gym Fetch Robotics tasks.

📄 PDF Abstract BibTeX arXiv:2110.02414

Code (0)

등록된 구현이 없습니다.

Tasks

FetchPush-v1Model-based Reinforcement LearningOpenAI Gym

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Exploration via Hindsight Goal Generation

2019-06-10 · NeurIPS 2019 12 · Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu 외

Goal-oriented reinforcement learning has recently been a practical framework for robotic manipulation tasks, in which an agent is required to reach a certain goal defined by a function on the state space. However, the sp…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Hindsight Curriculum Generation Based Multi-Goal Experience Replay

2021-01-01 · Xiaoyun Feng

In multi-goal tasks with sparse rewards, it is challenging to learn from tons of experiences with zero rewards. Hindsight experience replay (HER), which replays past experiences with additional heuristic goals, has shown…

Reinforcement Learning (RL)

Improvements on Hindsight Learning

2018-09-16 · Ameet Deshpande, Srikanth Sarma, Ashutosh Jha, Balaraman Ravindran

Sparse reward problems are one of the biggest challenges in Reinforcement Learning. Goal-directed tasks are one such sparse reward problems where a reward signal is received only when the goal is reached. One promising w…

Policy Gradient Methods reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Hindsight Experience Replay

2017-07-05 · NeurIPS 2017 12 · Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider 외

Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are s…

Reinforcement Learning Reinforcement Learning (RL)

DHER: Hindsight Experience Replay for Dynamic Goals

2019-05-01 · ICLR 2019 5 · Meng Fang, Cheng Zhou, Bei Shi, Boqing Gong 외

Dealing with sparse rewards is one of the most important challenges in reinforcement learning (RL), especially when a goal is dynamic (e.g., to grasp a moving object). Hindsight experience replay (HER) has been shown an …

Object Tracking Reinforcement Learning Reinforcement Learning (RL)