Curriculum-guided Hindsight Experience Replay

2019-12-01 · NeurIPS 2019 12 · Meng Fang, Tianyi Zhou, Yali Du, Lei Han, Zhengyou Zhang

In off-policy deep reinforcement learning, it is usually hard to collect sufficient successful experiences with sparse rewards to learn from. Hindsight experience replay (HER) enables an agent to learn from failures by treating the achieved state of a failed experience as a pseudo goal. However, not all the failed experiences are equally useful to different learning stages, so it is not efficient to replay all of them or uniform samples of them. In this paper, we propose to 1) adaptively select the failed experiences for replay according to the proximity to the true goals and the curiosity of exploration over diverse pseudo goals, and 2) gradually change the proportion of the goal-proximity and the diversity-based curiosity in the selection criteria: we adopt a human-like learning strategy that enforces more curiosity in earlier stages and changes to larger goal-proximity later. This `Goal-and-Curiosity-driven Curriculum Learning'' leads to `Curriculum-guided HER (CHER)'', which adaptively and dynamically controls the exploration-exploitation trade-off during the learning process via hindsight experience selection. We show that CHER improves the state of the art in challenging robotics environments.

📄 PDF Abstract BibTeX

Code (1)

mengf1/CHER 공식 구현 tf

Tasks

Deep Reinforcement LearningDiversityReinforcement Learning

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Hindsight Curriculum Generation Based Multi-Goal Experience Replay

2021-01-01 · Xiaoyun Feng

In multi-goal tasks with sparse rewards, it is challenging to learn from tons of experiences with zero rewards. Hindsight experience replay (HER), which replays past experiences with additional heuristic goals, has shown…

Reinforcement Learning (RL)

Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning

2021-02-25 · Rui Yang, Jiafei Lyu, Yu Yang, Jiangpeng Yan 외

Multi-goal reinforcement learning is widely applied in planning and robot manipulation. Two main challenges in multi-goal reinforcement learning are sparse rewards and sample inefficiency. Hindsight Experience Replay (HE…

Multi-Goal Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Curriculum Learning with Hindsight Experience Replay for Sequential Object Manipulation Tasks

2020-08-21 · Binyamin Manela, Armin Biess

Learning complex tasks from scratch is challenging and often impossible for humans as well as for artificial agents. A curriculum can be used instead, which decomposes a complex task (target task) into a sequence of sour…

Hindsight Experience Replay

2017-07-05 · NeurIPS 2017 12 · Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider 외

Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are s…

Reinforcement Learning Reinforcement Learning (RL)

Human-Aware Robot Navigation via Reinforcement Learning with Hindsight Experience Replay and Curriculum Learning

2021-10-09 · Keyu Li, Ye Lu, Max Q. -H. Meng

In recent years, the growing demand for more intelligent service robots is pushing the development of mobile robot navigation algorithms to allow safe and efficient operation in a dense crowd. Reinforcement learning (RL)…

Decision Making Reinforcement Learning (RL)Robot Navigation Sequential Decision Making