Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning

2022-06-21 · Haoqi Yuan, Zongqing Lu

We study offline meta-reinforcement learning, a practical reinforcement learning paradigm that learns from offline data to adapt to new tasks. The distribution of offline data is determined jointly by the behavior policy and the task. Existing offline meta-reinforcement learning algorithms cannot distinguish these factors, making task representations unstable to the change of behavior policies. To address this problem, we propose a contrastive learning framework for task representations that are robust to the distribution mismatch of behavior policies in training and test. We design a bi-level encoder structure, use mutual information maximization to formalize task representation learning, derive a contrastive learning objective, and introduce several approaches to approximate the true distribution of negative pairs. Experiments on a variety of offline meta-reinforcement learning benchmarks demonstrate the advantages of our method over prior methods, especially on the generalization to out-of-distribution behavior policies. The code is available at https://github.com/PKU-AI-Edge/CORRO.

📄 PDF Abstract BibTeX arXiv:2206.10442

Code (1)

pku-ai-edge/corro 공식 구현 pytorch

Tasks

Contrastive LearningMeta Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Representation Learning

Methods 이 논문이 사용한 방법론

Contrastive Learning 설명 없음

Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning

2021-02-22 · Lanqing Li, Yuanhao Huang, Mingzhe Chen, Siteng Luo 외

Meta-learning for offline reinforcement learning (OMRL) is an understudied problem with tremendous potential impact by enabling RL algorithms in many real-world applications. A popular solution to the problem is to infer…

Contrastive Learning Meta-Learning reinforcement-learning Reinforcement Learning (RL)+1

ContraBAR: Contrastive Bayes-Adaptive Deep RL

2023-06-04 · Era Choshen, Aviv Tamar

In meta reinforcement learning (meta RL), an agent seeks a Bayes-optimal policy -- the optimal policy when facing an unknown task that is sampled from some known task distribution. Previous approaches tackled this proble…

Contrastive Learning Meta Reinforcement Learning Variational Inference

Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations

2023-12-26 · Renzhe Zhou, Chen-Xiao Gao, Zongzhang Zhang, Yang Yu

Generalization and sample efficiency have been long-standing issues concerning reinforcement learning, and thus the field of Offline Meta-Reinforcement Learning~(OMRL) has gained increasing attention due to its potential…

Contrastive Learning Decoder Diversity Meta Reinforcement Learning+3

Contextual Latent World Models for Offline Meta Reinforcement Learning

2026-03-03 · Mohammadreza Nakheai, Aidan Scannell, Kevin Luck, Joni Pajarinen arxiv

Offline meta-reinforcement learning seeks to learn policies that generalize across related tasks from fixed datasets. Context-based methods infer a task representation from transition histories, but learning effective ta…

Representation Learning Reinforcement Learning

Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning

2024-12-19 · Mohammadreza Nakhaei, Aidan Scannell, Joni Pajarinen

Offline meta-reinforcement learning aims to equip agents with the ability to rapidly adapt to new tasks by training on data from a set of different tasks. Context-based approaches utilize a history of state-action-reward…

Meta Reinforcement Learning MuJoCo Representation Learning