Context-Specific Representation Abstraction for Deep Option Learning
Hierarchical reinforcement learning has focused on discovering temporally extended actions, such as options, that can provide benefits in problems requiring extensive exploration. One promising approach that learns these options end-to-end is the option-critic (OC) framework. We examine and show in this paper that OC does not decompose a problem into simpler sub-problems, but instead increases the size of the search over policy space with each option considering the entire state space during learning. This issue can result in practical limitations of this method, including sample inefficient learning. To address this problem, we introduce Context-Specific Representation Abstraction for Deep Option Learning (CRADOL), a new framework that considers both temporal abstraction and context-specific representation abstraction to effectively reduce the size of the search over policy space. Specifically, our method learns a factored belief state representation that enables each option to learn a policy over only a subsection of the state space. We test our method against hierarchical, non-hierarchical, and modular recurrent neural network baselines, demonstrating significant sample efficiency improvements in challenging partially observable environments.
Code (1)
Tasks
Hierarchical Reinforcement LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Abstraction for Offline Goal-Conditioned Reinforcement Learning
Markov Decision Processes (MDPs) often exhibit significant redundancy due to symmetries and shared structure across state-goal pairs in real-world Goal-Conditioned Reinforcement Learning (GCRL). While hierarchical polici…
Reinforcement LearningTemporal Abstraction in Reinforcement Learning with the Successor Representation
Reasoning at multiple levels of temporal abstraction is one of the key attributes of intelligence. In reinforcement learning, this is often modeled through temporally extended courses of actions called options. Options a…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Extracting Interpretable Local and Global Representations from Attention on Time Series
This paper targets two transformer attention based interpretability methods working with local abstraction and global representation, in the context of time series data. We distinguish local and global contexts, and prov…
Time SeriesOption Transfer and SMDP Abstraction with Successor Features
Abstraction plays an important role in the generalisation of knowledge and skills and is key to sample efficient learning. In this work, we study joint temporal and state abstraction in reinforcement learning, where temp…
Reinforcement Learning (RL)Abstraction for Deep Reinforcement Learning
We characterise the problem of abstraction in the context of deep reinforcement learning. Various well established approaches to analogical reasoning and associative memory might be brought to bear on this issue, but the…
BIG-bench Machine LearningDeep Reinforcement Learningreinforcement-learningReinforcement Learning+1