Latent Space Policies for Hierarchical Reinforcement Learning

2018-04-09 · ICML 2018 7 · Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, Sergey Levine

We address the problem of learning hierarchical deep neural network policies for reinforcement learning. In contrast to methods that explicitly restrict or cripple lower layers of a hierarchy to force them to use higher-level modulating signals, each layer in our framework is trained to directly solve the task, but acquires a range of diverse strategies via a maximum entropy reinforcement learning objective. Each layer is also augmented with latent random variables, which are sampled from a prior distribution during the training of that layer. The maximum entropy objective causes these latent variables to be incorporated into the layer's policy, and the higher level layer can directly control the behavior of the lower layer through this latent space. Furthermore, by constraining the mapping from latent variables to actions to be invertible, higher layers retain full expressivity: neither the higher layers nor the lower layers are constrained in their behavior. Our experimental evaluation demonstrates that we can improve on the performance of single-layer policies on standard benchmark tasks simply by adding additional layers, and that our method can solve more complex sparse-reward tasks by learning higher-level policies on top of high-entropy skills optimized for simple low-level objectives.

📄 PDF Abstract BibTeX arXiv:1804.02808

Code (0)

등록된 구현이 없습니다.

Tasks

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Disentangled Skill Embeddings for Reinforcement Learning

2019-06-21 · Janith C. Petangoda, Sergio Pascual-Diaz, Vincent Adam, Peter Vrancx 외

We propose a novel framework for multi-task reinforcement learning (MTRL). Using a variational inference formulation, we learn policies that generalize across both changing dynamics and goals. The resulting policies are …

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization

2019-01-05 · ICLR 2019 5 · Takayuki Osa, Voot Tangkaratt, Masashi Sugiyama

Real-world tasks are often highly structured. Hierarchical reinforcement learning (HRL) has attracted research interest as an approach for leveraging the hierarchical structure of a given task in reinforcement learning (…

continuous-control Continuous Control Diversity Hierarchical Reinforcement Learning+3

Wasserstein Unsupervised Reinforcement Learning

2021-10-15 · Shuncheng He, Yuhang Jiang, Hongchang Zhang, Jianzhun Shao 외

Unsupervised reinforcement learning aims to train agents to learn a handful of policies or skills in environments without external reward. These pre-trained policies can accelerate learning when endowed with external rew…

Hierarchical Reinforcement Learning MuJoCo reinforcement-learning Reinforcement Learning+2

Probabilistic Subgoal Representations for Hierarchical Reinforcement learning

2024-06-24 · Vivienne Huiling Wang, Tinghuai Wang, Wenyan Yang, Joni-Kristian Kämäräinen 외

In goal-conditioned hierarchical reinforcement learning (HRL), a high-level policy specifies a subgoal for the low-level policy to reach. Effective HRL hinges on a suitable subgoal represen tation function, abstracting s…

Gaussian Processes Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning

Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learning

2023-06-14 · Ce Hao, Catherine Weaver, Chen Tang, Kenta Kawamoto 외

Hierarchical reinforcement learning (RL) can accelerate long-horizon decision-making by temporally abstracting a policy into multiple levels. Promising results in sparse reward environments have been seen with skills, i.…

Autonomous Racing Decision Making Hierarchical Reinforcement Learning reinforcement-learning+2