Offline Meta Learning of Exploration

2020-08-06 · NeurIPS 2021 12 · Ron Dorfman, Idan Shenfeld, Aviv Tamar

Consider the following instance of the Offline Meta Reinforcement Learning (OMRL) problem: given the complete training logs of $N$ conventional RL agents, trained on $N$ different tasks, design a meta-agent that can quickly maximize reward in a new, unseen task from the same task distribution. In particular, while each conventional RL agent explored and exploited its own different task, the meta-agent must identify regularities in the data that lead to effective exploration/exploitation in the unseen task. Here, we take a Bayesian RL (BRL) view, and seek to learn a Bayes-optimal policy from the offline data. Building on the recent VariBAD BRL approach, we develop an off-policy BRL method that learns to plan an exploration strategy based on an adaptive neural belief estimate. However, learning to infer such a belief from offline data brings a new identifiability issue we term MDP ambiguity. We characterize the problem, and suggest resolutions via data collection and modification procedures. Finally, we evaluate our framework on a diverse set of domains, including difficult sparse reward tasks, and demonstrate learning of effective exploration behavior that is qualitatively different from the exploration used by any RL agent in the data.

📄 PDF Abstract BibTeX arXiv:2008.02598

Code (1)

Rondorf/BOReL 공식 구현 pytorch

Tasks

Meta-LearningMeta Reinforcement Learning

Offline Meta-Reinforcement Learning with Online Self-Supervision

2021-07-08 · Vitchyr H. Pong, Ashvin Nair, Laura Smith, Catherine Huang 외

Meta-reinforcement learning (RL) methods can meta-train policies that adapt to new tasks with orders of magnitude less data than standard RL, but meta-training itself is costly and time-consuming. If we can meta-train on…

Meta Reinforcement Learning Offline RL reinforcement-learning Reinforcement Learning+1

Offline Meta Reinforcement Learning -- Identifiability Challenges and Effective Data Collection Strategies

2021-05-21 · NeurIPS 2021 12 · Ron Dorfman, Idan Shenfeld, Aviv Tamar

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Context Shift Reduction for Offline Meta-Reinforcement Learning

2023-11-07 · NeurIPS 2023 11 · Yunkai Gao, Rui Zhang, Jiaming Guo, Fan Wu 외

Offline meta-reinforcement learning (OMRL) utilizes pre-collected offline datasets to enhance the agent's generalization ability on unseen tasks. However, the context shift problem arises due to the distribution discrepa…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Representation Learning

MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance

2021-12-07 · Michael Luo, Ashwin Balakrishna, Brijen Thananjeyan, Suraj Nair 외

Safe exploration is critical for using reinforcement learning (RL) in risk-sensitive environments. Recent work learns risk measures which measure the probability of violating constraints, which can then be used to enable…

continuous-control Continuous Control Meta-Learning Reinforcement Learning (RL)+1

Model-Based Offline Meta-Reinforcement Learning with Regularization

2022-02-07 · ICLR 2022 4 · Sen Lin, Jialin Wan, Tengyu Xu, Yingbin Liang 외

Existing offline reinforcement learning (RL) methods face a few major challenges, particularly the distributional shift between the learned policy and the behavior policy. Offline Meta-RL is emerging as a promising appro…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1