Provable Hierarchy-Based Meta-Reinforcement Learning
Hierarchical reinforcement learning (HRL) has seen widespread interest as an approach to tractable learning of complex modular behaviors. However, existing work either assume access to expert-constructed hierarchies, or use hierarchy-learning heuristics with no provable guarantees. To address this gap, we analyze HRL in the meta-RL setting, where a learner learns latent hierarchical structure during meta-training for use in a downstream task. We consider a tabular setting where natural hierarchical structure is embedded in the transition dynamics. Analogous to supervised meta-learning theory, we provide "diversity conditions" which, together with a tractable optimism-based algorithm, guarantee sample-efficient recovery of this natural hierarchy. Furthermore, we provide regret bounds on a learner using the recovered hierarchy to solve a meta-test task. Our bounds incorporate common notions in HRL literature such as temporal and state/action abstractions, suggesting that our setting and analysis capture important features of HRL in practice.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityHierarchical Reinforcement LearningLearning TheoryMeta-LearningMeta Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Constrained Meta Reinforcement Learning with Provable Test-Time Safety
Meta reinforcement learning (RL) allows agents to leverage experience across a distribution of tasks on which the agent can train at will, enabling faster learning of optimal policies on new test tasks. Despite its succe…
Reinforcement LearningMeta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator
Meta-reinforcement learning (Meta-RL) has attracted attention due to its capability to enhance reinforcement learning (RL) algorithms, in terms of data efficiency and generalizability. In this paper, we develop a bilevel…
AllBilevel OptimizationMeta Reinforcement Learningreinforcement-learning+2Cognitive Level-$k$ Meta-Learning for Safe and Pedestrian-Aware Autonomous Driving
The potential market for modern self-driving cars is enormous, as they are developing remarkably rapidly. At the same time, however, accidents of pedestrian fatalities caused by autonomous driving have been recorded in t…
Autonomous DrivingAutonomous VehiclesMeta-LearningMeta Reinforcement Learning+4Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
Reward decomposition is a critical problem in centralized training with decentralized execution~(CTDE) paradigm for multi-agent reinforcement learning. To take full advantage of global information, which exploits the sta…
Meta-LearningMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning (RL)+2Provable Safe Reinforcement Learning with Binary Feedback
Safety is a crucial necessity in many applications of reinforcement learning (RL), whether robotic, automotive, or medical. Many existing approaches to safe RL rely on receiving numeric safety feedback, but in many cases…
Active Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1