Wasserstein Unsupervised Reinforcement Learning
Unsupervised reinforcement learning aims to train agents to learn a handful of policies or skills in environments without external reward. These pre-trained policies can accelerate learning when endowed with external reward, and can also be used as primitive options in hierarchical reinforcement learning. Conventional approaches of unsupervised skill discovery feed a latent variable to the agent and shed its empowerment on agent's behavior by mutual information (MI) maximization. However, the policies learned by MI-based methods cannot sufficiently explore the state space, despite they can be successfully identified from each other. Therefore we propose a new framework Wasserstein unsupervised reinforcement learning (WURL) where we directly maximize the distance of state distributions induced by different policies. Additionally, we overcome difficulties in simultaneously training N(N >2) policies, and amortizing the overall reward to each step. Experiments show policies learned by our approach outperform MI-based methods on the metric of Wasserstein distance while keeping high discriminability. Furthermore, the agents trained by WURL can sufficiently explore the state space in mazes and MuJoCo tasks and the pre-trained policies can be applied to downstream tasks by hierarchical learning.
Code (0)
등록된 구현이 없습니다.
Tasks
Hierarchical Reinforcement LearningMuJoCoreinforcement-learningReinforcement LearningReinforcement Learning (RL)Unsupervised Reinforcement LearningSimilar Papers 제목 키워드 기반
Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning
Unsupervised reinforcement learning (URL) aims to learn general skills for unseen downstream tasks. Mutual Information Skill Learning (MISL) addresses URL by maximizing the mutual information between states and skills bu…
DisentanglementDiversityUnsupervised Reinforcement LearningWasserstein Dependency Measure for Representation Learning
Mutual information maximization has emerged as a powerful learning objective for unsupervised representation learning obtaining state-of-the-art performance in applications such as object recognition, speech recognition,…
Object Recognitionreinforcement-learningReinforcement LearningReinforcement Learning (RL)+4A proof of imitation of Wasserstein inverse reinforcement learning for multi-objective optimization
We prove Wasserstein inverse reinforcement learning enables the learner's reward values to imitate the expert's reward values in a finite iteration for multi-objective optimizations. Moreover, we prove Wasserstein invers…
reinforcement-learningReinforcement LearningUnsupervised Alignment of Embeddings with Wasserstein Procrustes
A library for Multilingual Unsupervised or Supervised word Embeddings
Word EmbeddingsReinforcement Learning with Wasserstein Distance Regularisation, with Applications to Multipolicy Learning
We describe an application of Wasserstein distance to Reinforcement Learning. The Wasserstein distance in question is between the distribution of mappings of trajectories of a policy into some metric space, and some othe…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)