Variational Curriculum Reinforcement Learning for Unsupervised Discovery of Skills
Mutual information-based reinforcement learning (RL) has been proposed as a promising framework for retrieving complex skills autonomously without a task-oriented reward function through mutual information (MI) maximization or variational empowerment. However, learning complex skills is still challenging, due to the fact that the order of training skills can largely affect sample efficiency. Inspired by this, we recast variational empowerment as curriculum learning in goal-conditioned RL with an intrinsic reward function, which we name Variational Curriculum RL (VCRL). From this perspective, we propose a novel approach to unsupervised skill discovery based on information theory, called Value Uncertainty Variational Curriculum (VUVC). We prove that, under regularity conditions, VUVC accelerates the increase of entropy in the visited states compared to the uniform curriculum. We validate the effectiveness of our approach on complex navigation and robotic manipulation tasks in terms of sample efficiency and state coverage speed. We also demonstrate that the skills discovered by our method successfully complete a real-world robot navigation task in a zero-shot setup and that incorporating these skills with a global planner further increases the performance.
Code (1)
Tasks
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Robot NavigationSimilar Papers 제목 키워드 기반
Unsupervised Skill-Discovery and Skill-Learning in Minecraft
Pre-training Reinforcement Learning agents in a task-agnostic manner has shown promising results. However, previous works still struggle in learning and discovering meaningful skills in high-dimensional state-spaces, suc…
MinecraftSelf-Supervised LearningDisentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning
A hallmark of intelligent agents is the ability to learn reusable skills purely from unsupervised interaction with the environment. However, existing unsupervised skill discovery methods often learn entangled skills wher…
DisentanglementHierarchical Reinforcement Learningreinforcement-learningReinforcement LearningA Unified Algorithm Framework for Unsupervised Discovery of Skills based on Determinantal Point Process
Learning rich skills under the option framework without supervision of external rewards is at the frontier of reinforcement learning research. Existing works mainly fall into two distinctive categories: variational optio…
Hierarchical Multi-Agent Skill Discovery
Skill discovery has shown significant progress in unsupervised reinforcement learning. This approach enables the discovery of a wide range of skills without any extrinsic reward, which can be effectively combined to tack…
Behavior Contrastive Learning for Unsupervised Skill Discovery
In reinforcement learning, unsupervised skill discovery aims to learn diverse skills without extrinsic rewards. Previous methods discover skills by maximizing the mutual information (MI) between states and skills. Howeve…
continuous-controlContinuous ControlContrastive Learning