Bayesian Policy Reuse

2015-05-01 · Benjamin Rosman, Majd Hawasly, Subramanian Ramamoorthy

A long-lived autonomous agent should be able to respond online to novel instances of tasks from a familiar domain. Acting online requires 'fast' responses, in terms of rapid convergence, especially when the task instance has a short duration, such as in applications involving interactions with humans. These requirements can be problematic for many established methods for learning to act. In domains where the agent knows that the task instance is drawn from a family of related tasks, albeit without access to the label of any given instance, it can choose to act through a process of policy reuse from a library, rather than policy learning from scratch. In policy reuse, the agent has prior knowledge of the class of tasks in the form of a library of policies that were learnt from sample task instances during an offline training phase. We formalise the problem of policy reuse, and present an algorithm for efficiently responding to a novel task instance by reusing a policy from the library of existing policies, where the choice is based on observed 'signals' which correlate to policy performance. We achieve this by posing the problem as a Bayesian choice problem with a corresponding notion of an optimal response, but the computation of that response is in many cases intractable. Therefore, to reduce the computation cost of the posterior, we follow a Bayesian optimisation approach and define a set of policy selection functions, which balance exploration in the policy library against exploitation of previously tried policies, together with a model of expected performance of the policy library on their corresponding task instances. We validate our method in several simulated domains of interactive, short-duration episodic tasks, showing rapid convergence in unknown task variations.

📄 PDF Abstract BibTeX arXiv:1505.00284

Code (0)

등록된 구현이 없습니다.

Tasks

Bayesian Optimisation

Efficient Bayesian Policy Reuse with a Scalable Observation Model in Deep Reinforcement Learning

2022-04-16 · Jinmei Liu, Zhi Wang, Chunlin Chen, Daoyi Dong

Bayesian policy reuse (BPR) is a general policy transfer framework for selecting a source policy from an offline library by inferring the task belief based on some observation signals and a trained observation model. In …

Continual Learning Deep Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

A Deep Bayesian Policy Reuse Approach Against Non-Stationary Agents

2018-12-01 · NeurIPS 2018 12 · Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang 외

In multiagent domains, coping with non-stationary agents that change behaviors from time to time is a challenging problem, where an agent is usually required to be able to quickly detect the other agent's policy during o…

Context-Aware Policy Reuse

2018-06-11 · Siyuan Li, Fangda Gu, Guangxiang Zhu, Chongjie Zhang

Transfer learning can greatly speed up reinforcement learning for a new task by leveraging policies of relevant tasks. Existing works of policy reuse either focus on only selecting a single best source policy for trans…

Reinforcement Learning Transfer Learning

CUP: Critic-Guided Policy Reuse

2022-10-15 · Jin Zhang, Siyuan Li, Chongjie Zhang

The ability to reuse previous policies is an important aspect of human intelligence. To achieve efficient policy reuse, a Deep Reinforcement Learning (DRL) agent needs to decide when to reuse and which source policies to…

Deep Reinforcement Learning

Lifetime policy reuse and the importance of task capacity

2021-06-03 · David M. Bossens, Adam J. Sobey

A long-standing challenge in artificial intelligence is lifelong reinforcement learning, where learners are given many tasks in sequence and must transfer knowledge between tasks while avoiding catastrophic forgetting. P…

reinforcement-learning Reinforcement Learning