A Laplacian Framework for Option Discovery in Reinforcement Learning

2017-03-02 · ICML 2017 8 · Marlos C. Machado, Marc G. Bellemare, Michael Bowling

Representation learning and option discovery are two of the biggest challenges in reinforcement learning (RL). Proto-value functions (PVFs) are a well-known approach for representation learning in MDPs. In this paper we address the option discovery problem by showing how PVFs implicitly define options. We do it by introducing eigenpurposes, intrinsic reward functions derived from the learned representations. The options discovered from eigenpurposes traverse the principal directions of the state space. They are useful for multiple tasks because they are discovered without taking the environment's rewards into consideration. Moreover, different options act at different time scales, making them helpful for exploration. We demonstrate features of eigenpurposes in traditional tabular domains as well as in Atari 2600 games.

📄 PDF Abstract BibTeX arXiv:1703.00956

Code (1)

mcmachado/options 공식 구현

Tasks

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Representation Learning

A Unified Algorithm Framework for Unsupervised Discovery of Skills based on Determinantal Point Process

2023-09-21 · NeurIPS 2023 11

Learning rich skills under the option framework without supervision of external rewards is at the frontier of reinforcement learning research. Existing works mainly fall into two distinctive categories: variational optio…

Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing

2021-07-12 · Kaixin Wang, Kuangqi Zhou, Qixin Zhang, Jie Shao 외

The Laplacian representation recently gains increasing attention for reinforcement learning as it provides succinct and informative representation for states, by taking the eigenvectors of the Laplacian matrix of the sta…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning (RL)

Deep Laplacian-based Options for Temporally-Extended Exploration

2023-01-26 · Martin Klissarov, Marlos C. Machado

Selecting exploratory actions that generate a rich stream of experience for better learning is a fundamental challenge in reinforcement learning (RL). An approach to tackle this problem consists in selecting actions acco…

Reinforcement Learning (RL)

Option Discovery in the Absence of Rewards with Manifold Analysis

2020-03-12 · ICML 2020 1 · Amitay Bar, Ronen Talmon, Ron Meir

Options have been shown to be an effective tool in reinforcement learning, facilitating improved exploration and learning. In this paper, we present an approach based on spectral graph theory and derive an algorithm that…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Exploration in Reinforcement Learning with Deep Covering Options

2020-05-01 · ICLR 2020 1 · Yuu Jinnai, Jee Won Park, Marlos C. Machado, George Konidaris

While many option discovery methods have been proposed to accelerate exploration in reinforcement learning, they are often heuristic. Recently, covering options was proposed to discover a set of options that provably red…