Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning
Practising and honing skills forms a fundamental component of how humans learn, yet artificial agents are rarely specifically trained to perform them. Instead, they are usually trained end-to-end, with the hope being that useful skills will be implicitly learned in order to maximise discounted return of some extrinsic reward function. In this paper, we investigate how skills can be incorporated into the training of reinforcement learning (RL) agents in complex environments with large state-action spaces and sparse rewards. To this end, we created SkillHack, a benchmark of tasks and associated skills based on the game of NetHack. We evaluate a number of baselines on this benchmark, as well as our own novel skill-based method Hierarchical Kickstarting (HKS), which is shown to outperform all other evaluated methods. Our experiments show that learning with a prior knowledge of useful skills can significantly improve the performance of agents on complex problems. We ultimately argue that utilising predefined skills provides a useful inductive bias for RL problems, especially those with large state-action spaces and sparse rewards.
Code (2)
Tasks
Inductive BiasNetHackreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Kickstarting Deep Reinforcement Learning
We present a method for using previously-trained 'teacher' agents to kickstart the training of a new 'student' agent. To this end, we leverage ideas from policy distillation and population based training. Our method plac…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Privacy-Preserving Kickstarting Deep Reinforcement Learning with Privacy-Aware Learners
Kickstarting deep reinforcement learning algorithms facilitate a teacher-student relationship among the agents and allow for a well-performing teacher to share demonstrations with a student to expedite the student's trai…
Deep Reinforcement LearningPrivacy Preservingreinforcement-learningReinforcement Learning+1Sub-policy Adaptation for Hierarchical Reinforcement Learning
Hierarchical reinforcement learning is a promising approach to tackle long-horizon decision-making problems with sparse rewards. Unfortunately, most methods still decouple the lower-level skill acquisition process and th…
Decision MakingHierarchical Reinforcement Learningreinforcement-learningReinforcement Learning+1Sub-policy Adaptation for Hierarchical Reinforcement Learning
Hierarchical Reinforcement Learning is a promising approach to long-horizon decision-making problems with sparse rewards. Unfortunately, most methods still decouple the lower-level skill acquisition process and the train…
Decision MakingHierarchical Reinforcement Learningreinforcement-learningReinforcement Learning+1Composing Task-Agnostic Policies with Deep Reinforcement Learning
The composition of elementary behaviors to solve challenging transfer learning problems is one of the key elements in building intelligent machines. To date, there has been plenty of work on learning task-specific polici…
Decision MakingDeep Reinforcement LearningMotion Planningreinforcement-learning+4