Meta-Gradient Reinforcement Learning with an Objective Discovered Online

2020-07-16 · NeurIPS 2020 12 · Zhongwen Xu, Hado van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, David Silver

Deep reinforcement learning includes a broad family of algorithms that parameterise an internal representation, such as a value function or policy, by a deep neural network. Each algorithm optimises its parameters with respect to an objective, such as Q-learning or policy gradient, that defines its semantics. In this work, we propose an algorithm based on meta-gradient descent that discovers its own objective, flexibly parameterised by a deep neural network, solely from interactive experience with its environment. Over time, this allows the agent to learn how to learn increasingly effectively. Furthermore, because the objective is discovered online, it can adapt to changes over time. We demonstrate that the algorithm discovers how to address several important issues in RL, such as bootstrapping, non-stationarity, and off-policy learning. On the Atari Learning Environment, the meta-gradient algorithm adapts over time to learn with greater efficiency, eventually outperforming the median score of a strong actor-critic baseline.

📄 PDF Abstract BibTeX arXiv:2007.08433

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement LearningQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Discovering Temporally-Aware Reinforcement Learning Algorithms

2024-02-08 · Matthew Thomas Jackson, Chris Lu, Louis Kirsch, Robert Tjarko Lange 외

Recent advancements in meta-learning have enabled the automatic discovery of novel reinforcement learning algorithms parameterized by surrogate objective functions. To improve upon manually designed algorithms, the param…

Meta-Learning reinforcement-learning Reinforcement Learning

Debiasing Meta-Gradient Reinforcement Learning by Learning the Outer Value Function

2022-11-19 · Clément Bonnet, Laurence Midgley, Alexandre Laterre

Meta-gradient Reinforcement Learning (RL) allows agents to self-tune their hyper-parameters in an online fashion during training. In this paper, we identify a bias in the meta-gradient of current meta-gradient RL approac…

reinforcement-learning Reinforcement Learning (RL)

Discovery of Options via Meta-Learned Subgoals

2021-02-12 · NeurIPS 2021 12 · Vivek Veeriah, Tom Zahavy, Matteo Hessel, Zhongwen Xu 외

Temporal abstractions in the form of options have been shown to help reinforcement learning (RL) agents learn faster. However, despite prior work on this topic, the problem of discovering options through interaction with…

Reinforcement Learning (RL)

Improving Generalization in Meta Reinforcement Learning using Learned Objectives

2019-10-09 · ICLR 2020 1 · Louis Kirsch, Sjoerd van Steenkiste, Jürgen Schmidhuber

Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans. Our novel meta reinforcement learning algorithm MetaGenRL is inspired by this process. MetaGenRL distill…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning

2021-08-30 · Shenao Zhang, Lei Han, Li Shen

In multi-agent reinforcement learning, the behaviors that agents learn in a single Markov Game (MG) are typically confined to the given agent number. Every single MG induced by varying the population may possess distinct…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)