Relay Hindsight Experience Replay: Self-Guided Continual Reinforcement Learning for Sequential Object Manipulation Tasks with Sparse Rewards

2022-08-01 · Yongle Luo, Yuxin Wang, Kun Dong, Qiang Zhang, Erkang Cheng, Zhiyong Sun, Bo Song

Exploration with sparse rewards remains a challenging research problem in reinforcement learning (RL). Especially for sequential object manipulation tasks, the RL agent always receives negative rewards until completing all sub-tasks, which results in low exploration efficiency. To solve these tasks efficiently, we propose a novel self-guided continual RL framework, RelayHER (RHER). RHER first decomposes a sequential task into new sub-tasks with increasing complexity and ensures that the simplest sub-task can be learned quickly by utilizing Hindsight Experience Replay (HER). Secondly, we design a multi-goal & multi-task network to learn these sub-tasks simultaneously. Finally, we propose a Self-Guided Exploration Strategy (SGES). With SGES, the learned sub-task policy will guide the agent to the states that are helpful to learn more complex sub-task with HER. By this self-guided exploration and relay policy learning, RHER can solve these sequential tasks efficiently stage by stage. The experimental results show that RHER significantly outperforms vanilla-HER in sample-efficiency on five singleobject and five complex multi-object manipulation tasks (e.g., Push, Insert, ObstaclePush, Stack, TStack, etc.). The proposed RHER has also been applied to learn a contact-rich push task on a physical robot from scratch, and the success rate reached 10/10 with only 250 episodes.

📄 PDF Abstract BibTeX arXiv:2208.00843

Code (1)

kaixindelele/RHER 공식 구현 tf

Tasks

FetchPush-v1Reinforcement Learning (RL)Robot Manipulation

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

MRHER: Model-based Relay Hindsight Experience Replay for Sequential Object Manipulation Tasks with Sparse Rewards

2023-06-28 · Yuming Huang, Bin Ren, Ziming Xu, Lianghong Wu

Sparse rewards pose a significant challenge to achieving high sample efficiency in goal-conditioned reinforcement learning (RL). Specifically, in sequential manipulation tasks, the agent receives failure rewards until it…

FetchPush-v1 Multi-Goal Reinforcement Learning Reinforcement Learning (RL)Robot Manipulation

Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment

2021-10-28 · Tung M. Luu, Chang D. Yoo

This paper proposes a method for prioritizing the replay experience referred to as Hindsight Goal Ranking (HGR) in overcoming the limitation of Hindsight Experience Replay (HER) that generates hindsight goals based on un…

Curriculum-guided Hindsight Experience Replay

2019-12-01 · NeurIPS 2019 12 · Meng Fang, Tianyi Zhou, Yali Du, Lei Han 외

In off-policy deep reinforcement learning, it is usually hard to collect sufficient successful experiences with sparse rewards to learn from. Hindsight experience replay (HER) enables an agent to learn from failures by t…

Deep Reinforcement Learning Diversity Reinforcement Learning

Hindsight Curriculum Generation Based Multi-Goal Experience Replay

2021-01-01 · Xiaoyun Feng

In multi-goal tasks with sparse rewards, it is challenging to learn from tons of experiences with zero rewards. Hindsight experience replay (HER), which replays past experiences with additional heuristic goals, has shown…

Reinforcement Learning (RL)

GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning

2025-08-08 · Xing Lei, Wenyan Yang, Kaiqiang Ke, Shentao Yang 외 arxiv

Goal-conditioned reinforcement learning (GCRL) with sparse rewards remains a fundamental challenge in reinforcement learning. While hindsight experience replay (HER) has shown promise by relabeling collected trajectories…

Reinforcement Learning