Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations
Currently, deep reinforcement learning (RL) shows impressive results in complex gaming and robotic environments. Often these results are achieved at the expense of huge computational costs and require an incredible number of episodes of interaction between the agent and the environment. There are two main approaches to improving the sample efficiency of reinforcement learning methods - using hierarchical methods and expert demonstrations. In this paper, we propose a combination of these approaches that allow the agent to use low-quality demonstrations in complex vision-based environments with multiple related goals. Our forgetful experience replay (ForgER) algorithm effectively handles errors in expert data and reduces quality losses when adapting the action space and states representation to the agent's capabilities. Our proposed goal-oriented structuring of replay buffer allows the agent to automatically highlight sub-goals for solving complex hierarchical tasks in demonstrations. Our method is universal and can be integrated into various off-policy methods. It surpasses all known existing state-of-the-art RL methods using expert demonstrations on various model environments. The solution based on our algorithm beats all the solutions for the famous MineRL competition and allows the agent to mine a diamond in the Minecraft environment.
Code (1)
Tasks
Deep Reinforcement LearningHierarchical Reinforcement LearningMinecraftreinforcement-learningReinforcement LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Mixing Human Demonstrations with Self-Exploration in Experience Replay for Deep Reinforcement Learning
We investigate the effect of using human demonstration data in the replay buffer for Deep Reinforcement Learning. We use a policy gradient method with a modified experience replay buffer where a human demonstration exper…
Deep Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Dynamic Experience Replay
We present a novel technique called Dynamic Experience Replay (DER) that allows Reinforcement Learning (RL) algorithms to use experience replay samples not only from human demonstrations but also successful transitions g…
Reinforcement LearningReinforcement Learning (RL)Learning Memory-Dependent Continuous Control from Demonstrations
Efficient exploration has presented a long-standing challenge in reinforcement learning, especially when rewards are sparse. A developmental system can overcome this difficulty by learning from both demonstrations and se…
continuous-controlContinuous ControlDecision MakingEfficient Exploration+3Designing Less Forgetful Networks for Continual Learning
Neural networks usually excel in learning a single task. Their weights are plastic and help them to learn quickly, but these weights are also known to be unstable. Hence, they may experience catastrophic forgetting and l…
Continual LearningSample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraft
Sample inefficiency of deep reinforcement learning methods is a major obstacle for their use in real-world applications. In this work, we show how human demonstrations can improve final performance of agents on the Minec…
Deep Reinforcement LearningMinecraftreinforcement-learningReinforcement Learning+1