Risk-Aware Transfer in Reinforcement Learning using Successor Features

2021-05-28 · NeurIPS 2021 12 · Michael Gimelfarb, André Barreto, Scott Sanner, Chi-Guhn Lee

Sample efficiency and risk-awareness are central to the development of practical reinforcement learning (RL) for complex decision-making. The former can be addressed by transfer learning and the latter by optimizing some utility function of the return. However, the problem of transferring skills in a risk-aware manner is not well-understood. In this paper, we address the problem of risk-aware policy transfer between tasks in a common domain that differ only in their reward functions, in which risk is measured by the variance of reward streams. Our approach begins by extending the idea of generalized policy improvement to maximize entropic utilities, thus extending policy improvement via dynamic programming to sets of policies and levels of risk-aversion. Next, we extend the idea of successor features (SF), a value function representation that decouples the environment dynamics from the rewards, to capture the variance of returns. Our resulting risk-aware successor features (RaSF) integrate seamlessly within the RL framework, inherit the superior task generalization ability of SFs, and incorporate risk-awareness into the decision-making. Experiments on a discrete navigation domain and control of a simulated robotic arm demonstrate the ability of RaSFs to outperform alternative methods including SFs, when taking the risk of the learned policies into account.

📄 PDF Abstract BibTeX arXiv:2105.14127

Code (0)

등록된 구현이 없습니다.

Tasks

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Transfer Learning

Advantages and Limitations of using Successor Features for Transfer in Reinforcement Learning

2017-07-31 · Lucas Lehnert, Stefanie Tellex, Michael L. Littman

One question central to Reinforcement Learning is how to learn a feature representation that supports algorithm scaling and re-use of learned information from different tasks. Successor Features approach this problem by …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A New Representation of Successor Features for Transfer across Dissimilar Environments

2021-07-18 · Majid Abdolshah, Hung Le, Thommen Karimpanal George, Sunil Gupta 외

Transfer in reinforcement learning is usually achieved through generalisation across tasks. Whilst many studies have investigated transferring knowledge when the reward function changes, they have assumed that the dynami…

Gaussian Processes Reinforcement Learning (RL)

Successor Feature Representations

2021-10-29 · Chris Reinke, Xavier Alameda-Pineda

Transfer in Reinforcement Learning aims to improve learning performance on target tasks using knowledge from experienced source tasks. Successor Representations (SR) and their extension Successor Features (SF) are promin…

Transfer Learning

Universal Successor Features for Transfer Reinforcement Learning

2020-01-05 · ICLR 2019 5 · Chen Ma, Dylan R. Ashley, Junfeng Wen, Yoshua Bengio

Transfer in Reinforcement Learning (RL) refers to the idea of applying knowledge gained from previous tasks to solving related tasks. Learning a universal value function (Schaul et al., 2015), which generalizes over goal…

MuJoCo reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Transfer with Model Features in Reinforcement Learning

2018-07-04 · Lucas Lehnert, Michael L. Littman

A key question in Reinforcement Learning is which representation an agent can learn to efficiently reuse knowledge between different tasks. Recently the Successor Representation was shown to have empirical benefits for t…

model reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)