Matching options to tasks using Option-Indexed Hierarchical Reinforcement Learning

2022-06-12 · Kushal Chauhan, Soumya Chatterjee, Akash Reddy, Balaraman Ravindran, Pradeep Shenoy

The options framework in Hierarchical Reinforcement Learning breaks down overall goals into a combination of options or simpler tasks and associated policies, allowing for abstraction in the action space. Ideally, these options can be reused across different higher-level goals; indeed, such reuse is necessary to realize the vision of a continual learning agent that can effectively leverage its prior experience. Previous approaches have only proposed limited forms of transfer of prelearned options to new task settings. We propose a novel option indexing approach to hierarchical learning (OI-HRL), where we learn an affinity function between options and the items present in the environment. This allows us to effectively reuse a large library of pretrained options, in zero-shot generalization at test time, by restricting goal-directed learning to only those options relevant to the task at hand. We develop a meta-training loop that learns the representations of options and environments over a series of HRL problems, by incorporating feedback about the relevance of retrieved options to the higher-level goal. We evaluate OI-HRL in two simulated settings - the CraftWorld and AI2THOR environments - and show that we achieve performance competitive with oracular baselines, and substantial gains over a baseline that has the entire option pool available for learning the hierarchical policy.

📄 PDF Abstract BibTeX arXiv:2206.05750

Code (0)

등록된 구현이 없습니다.

Tasks

Continual Learning Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Zero-shot Generalization

Methods 이 논문이 사용한 방법론

Test 설명 없음

Hierarchical Reinforcement Learning By Discovering Intrinsic Options

2021-01-16 · ICLR 2021 1 · Jesse Zhang, Haonan Yu, Wei Xu

We propose a hierarchical reinforcement learning method, HIDIO, that can learn task-agnostic options in a self-supervised manner while jointly learning to utilize them to solve sparse-reward tasks. Unlike current hierarc…

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Soft Options Critic

2019-05-23 · Elita Lobo, Scott Jordan

The option-critic architecture (Bacon, Harb, and Precup 2017) and several variants have successfully demonstrated the use of the options framework proposed by Sutton et al (Sutton, Precup, and Singh1999) to scale learnin…

Multitask Soft Option Learning

2019-04-01 · Maximilian Igl, Andrew Gambardella, Jinke He, Nantas Nardelli 외

We present Multitask Soft Option Learning(MSOL), a hierarchical multitask framework based on Planning as Inference. MSOL extends the concept of options, using separate variational posteriors for each task, regularized by…

Transfer Learning

Learning to Drive Safely with Hybrid Options

2025-10-28 · Bram De Cooman, Johan Suykens arxiv

Out of the many deep reinforcement learning approaches for autonomous driving, only few make use of the options (or skills) framework. That is surprising, as this framework is naturally suited for hierarchical control ap…

Reinforcement Learning Autonomous Driving

Meta Reinforcement Learning for Fast Adaptation of Hierarchical Policies

2021-05-21 · NeurIPS 2021 12 · David Kuric, Herke van Hoof

Hierarchical methods have the potential to allow reinforcement learning to scale to larger environments. Decomposing a task into transferable components, however, remains a challenging problem. In this paper, we propose …

Meta-Learning Meta Reinforcement Learning reinforcement-learning Reinforcement Learning+1