Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation
Complex sequential tasks in continuous-control settings often require agents to successfully traverse a set of "narrow passages" in their state space. Solving such tasks with a sparse reward in a sample-efficient manner poses a challenge to modern reinforcement learning (RL) due to the associated long-horizon nature of the problem and the lack of sufficient positive signal during learning. Various tools have been applied to address this challenge. When available, large sets of demonstrations can guide agent exploration. Hindsight relabelling on the other hand does not require additional sources of information. However, existing strategies explore based on task-agnostic goal distributions, which can render the solution of long-horizon tasks impractical. In this work, we extend hindsight relabelling mechanisms to guide exploration along task-specific distributions implied by a small set of successful demonstrations. We evaluate the approach on four complex, single and dual arm, robotics manipulation tasks against strong suitable baselines. The method requires far fewer demonstrations to solve all tasks and achieves a significantly higher overall performance as task complexity increases. Finally, we investigate the robustness of the proposed solution with respect to the quality of input representations and the number of demonstrations.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Hindsight policy gradients
A reinforcement learning agent that needs to pursue different goals across episodes requires a goal-conditional policy. In addition to their potential to generalize desirable behavior to unseen goals, such policies may a…
Policy Gradient Methodsreinforcement-learningReinforcement LearningReinforcement Learning (RL)Curriculum-guided Hindsight Experience Replay
In off-policy deep reinforcement learning, it is usually hard to collect sufficient successful experiences with sparse rewards to learn from. Hindsight experience replay (HER) enables an agent to learn from failures by t…
Deep Reinforcement LearningDiversityReinforcement LearningDiversity-based Trajectory and Goal Selection with Hindsight Experience Replay
Hindsight experience replay (HER) is a goal relabelling technique typically used with off-policy deep reinforcement learning algorithms to solve goal-oriented tasks; it is well suited to robotic manipulation tasks that d…
Deep Reinforcement LearningDiversityPoint ProcessesApproachability in unknown games: Online learning meets multi-objective optimization
In the standard setting of approachability there are two players and a target set. The players play repeatedly a known vector-valued game where the first player wants to have the average vector-valued payoff converge to …
Improvements on Hindsight Learning
Sparse reward problems are one of the biggest challenges in Reinforcement Learning. Goal-directed tasks are one such sparse reward problems where a reward signal is received only when the goal is reached. One promising w…
Policy Gradient Methodsreinforcement-learningReinforcement LearningReinforcement Learning (RL)