Model-Based Reinforcement Learning with Multi-Task Offline Pretraining

2023-06-06 · Minting Pan, Yitao Zheng, Yunbo Wang, Xiaokang Yang

Pretraining reinforcement learning (RL) models on offline datasets is a promising way to improve their training efficiency in online tasks, but challenging due to the inherent mismatch in dynamics and behaviors across various tasks. We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task. The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance for both dynamics representation transfer and policy transfer. We build a time-varying, domain-selective distillation loss to generate a set of offline-to-online similarity weights. These weights serve two purposes: (i) adaptively transferring the task-agnostic knowledge of physical dynamics to facilitate world model training, and (ii) learning to replay relevant source actions to guide the target policy. We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.

📄 PDF Abstract BibTeX arXiv:2306.03360

Code (1)

panmt/vid2act 공식 구현 pytorch

Tasks

Knowledge DistillationModel-based Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning

2026-08-11 · Daoyi Li, Yixian Zhang, Wenbo Ding, Yu Wang 외 arxiv

Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder onlin…

Reinforcement Learning

Guiding Online Reinforcement Learning with Action-Free Offline Pretraining

2023-01-30 · Deyao Zhu, Yuhui Wang, Jürgen Schmidhuber, Mohamed Elhoseiny

Offline RL methods have been shown to reduce the need for environment interaction by training agents using offline collected episodes. However, these methods typically require action information to be logged during data …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Representation Matters: Offline Pretraining for Sequential Decision Making

2021-02-11 · ICLR Workshop SSL-RL 2021 5 · Mengjiao Yang, Ofir Nachum

The recent success of supervised learning methods on ever larger offline datasets has spurred interest in the reinforcement learning (RL) field to investigate whether the same paradigms can be translated to RL algorithms…

Decision Making Imitation Learning Offline RL Reinforcement Learning (RL)+1

Contextual Transformer for Offline Meta Reinforcement Learning

2022-11-15 · Runji Lin, Ye Li, Xidong Feng, Zhaowei Zhang 외

The pretrain-finetuning paradigm in large-scale sequence models has made significant progress in natural language processing and computer vision tasks. However, such a paradigm is still hindered by several challenges in …

D4RL Meta Reinforcement Learning MuJoCo Offline RL+5

Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning

2025-05-09 · Jongchan Park, MinGyu Park, Donghwan Lee

Offline reinforcement learning (RL) aims to learn a policy from a static dataset without further interactions with the environment. Collecting sufficiently large datasets for offline RL is exhausting since this data coll…

D4RL Offline RL Reinforcement Learning (RL)