Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards

2019-10-10 · NeurIPS 2019 12 · Siyuan Li, Rui Wang, Minxue Tang, Chongjie Zhang

Hierarchical Reinforcement Learning (HRL) is a promising approach to solving long-horizon problems with sparse and delayed rewards. Many existing HRL algorithms either use pre-trained low-level skills that are unadaptable, or require domain-specific information to define low-level rewards. In this paper, we aim to adapt low-level skills to downstream tasks while maintaining the generality of reward design. We propose an HRL framework which sets auxiliary rewards for low-level skill training based on the advantage function of the high-level policy. This auxiliary reward enables efficient, simultaneous learning of the high-level policy and low-level skills without using task-specific knowledge. In addition, we also theoretically prove that optimizing low-level skills with this auxiliary reward will increase the task return for the joint policy. Experimental results show that our algorithm dramatically outperforms other state-of-the-art HRL methods in Mujoco domains. We also find both low-level and high-level policies trained by our algorithm transferable.

📄 PDF Abstract BibTeX arXiv:1910.04450

Code (1)

ArayCHN/HAAR-A-Hierarchical-RL-Algorithm

Tasks

Hierarchical Reinforcement LearningMuJoCoreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

2017-05-18 · Nat Dilokthanakul, Christos Kaplanis, Nick Pawlowski, Murray Shanahan

The problem of sparse rewards is one of the hardest challenges in contemporary reinforcement learning. Hierarchical reinforcement learning (HRL) tackles this problem by using a set of temporally-extended actions, or opti…

Hierarchical Reinforcement Learning Montezuma's Revenge reinforcement-learning Reinforcement Learning+1

Bayesian Hierarchical Reinforcement Learning

2012-12-01 · NeurIPS 2012 12 · Feng Cao, Soumya Ray

We describe an approach to incorporating Bayesian priors in the maxq framework for hierarchical reinforcement learning (HRL). We define priors on the primitive environment model and on task pseudo-rewards. Since models f…

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Follow your Nose: Using General Value Functions for Directed Exploration in Reinforcement Learning

2022-03-02 · Durgesh Kalwar, Omkar Shelke, Somjit Nath, Hardik Meisheri 외

Improving sample efficiency is a key challenge in reinforcement learning, especially in environments with large state spaces and sparse rewards. In literature, this is resolved either through the use of auxiliary tasks (…

reinforcement-learning Reinforcement Learning (RL)

Agent Modeling as Auxiliary Task for Deep Reinforcement Learning

2019-07-22 · Pablo Hernandez-Leal, Bilal Kartal, Matthew E. Taylor

In this paper we explore how actor-critic methods in deep reinforcement learning, in particular Asynchronous Advantage Actor-Critic (A3C), can be extended with agent modeling. Inspired by recent works on representation l…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Reward Training Wheels: Adaptive Auxiliary Rewards for Robotics Reinforcement Learning

2025-03-19 · Linji Wang, Tong Xu, Yuanjie Lu, Xuesu Xiao

Robotics Reinforcement Learning (RL) often relies on carefully engineered auxiliary rewards to supplement sparse primary learning objectives to compensate for the lack of large-scale, real-world, trial-and-error data. Wh…

Reinforcement Learning (RL)