OptionZero: Planning with Learned Options
Planning with options -- a sequence of primitive actions -- has been shown effective in reinforcement learning within complex environments. Previous studies have focused on planning with predefined options or learned options through expert demonstration data. Inspired by MuZero, which learns superhuman heuristics without any human knowledge, we propose a novel approach, named OptionZero. OptionZero incorporates an option network into MuZero, providing autonomous discovery of options through self-play games. Furthermore, we modify the dynamics network to provide environment transitions when using options, allowing searching deeper under the same simulation constraints. Empirical experiments conducted in 26 Atari games demonstrate that OptionZero outperforms MuZero, achieving a 131.58% improvement in mean human-normalized score. Our behavior analysis shows that OptionZero not only learns options but also acquires strategic skills tailored to different game characteristics. Our findings show promising directions for discovering and using options in planning. Our code is available at https://rlg.iis.sinica.edu.tw/papers/optionzero.
Code (1)
Tasks
Atari GamesMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GrASP: Gradient-Based Affordance Selection for Planning
Planning with a learned model is arguably a key component of intelligence. There are several challenges in realizing such a component in large-scale reinforcement learning (RL) problems. One such challenge is dealing eff…
Reinforcement Learning (RL)Reward-Respecting Subtasks for Model-Based Reinforcement Learning
To achieve the ambitious goals of artificial intelligence, reinforcement learning must include planning with a model of the world that is abstract in state and time. Deep learning has made progress with state abstraction…
Model-based Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Iterative Option Discovery for Planning, by Planning
Discovering useful temporal abstractions, in the form of options, is widely thought to be key to applying reinforcement learning and planning to increasingly complex domains. Building on the empirical success of the Expe…
Creativity of AI: Hierarchical Planning Model Learning for Facilitating Deep Reinforcement Learning
Despite of achieving great success in real-world applications, Deep Reinforcement Learning (DRL) is still suffering from three critical issues, i.e., data efficiency, lack of the interpretability and transferability. Rec…
Deep Reinforcement LearningMontezuma's Revengereinforcement-learningReinforcement Learning+1Learning Options from Demonstration using Skill Segmentation
We present a method for learning options from segmented demonstration trajectories. The trajectories are first segmented into skills using nonparametric Bayesian clustering and a reward function for each segment is then …
Clusteringreinforcement-learningReinforcement LearningReinforcement Learning (RL)