Policy Expansion for Bridging Offline-to-Online Reinforcement Learning

2023-02-02 · Haichao Zhang, We Xu, Haonan Yu

Pre-training with offline data and online fine-tuning using reinforcement learning is a promising strategy for learning control policies by leveraging the best of both worlds in terms of sample efficiency and performance. One natural approach is to initialize the policy for online learning with the one trained offline. In this work, we introduce a policy expansion scheme for this task. After learning the offline policy, we use it as one candidate policy in a policy set. We then expand the policy set with another policy which will be responsible for further learning. The two policies will be composed in an adaptive manner for interacting with the environment. With this approach, the policy previously learned offline is fully retained during online learning, thus mitigating the potential issues such as destroying the useful behaviors of the offline policy in the initial stage of online learning while allowing the offline policy participate in the exploration naturally in an adaptive manner. Moreover, new useful behaviors can potentially be captured by the newly added policy through learning. Experiments are conducted on a number of tasks and the results demonstrate the effectiveness of the proposed approach.

📄 PDF Abstract BibTeX arXiv:2302.00935

Code (1)

haichao-zhang/pex 공식 구현 pytorch

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning

2024-10-31 · Jaeyoon Kim, Junyu Xuan, Christy Liang, Farookh Hussain

Offline-to-online reinforcement learning (RL) leverages both pre-trained offline policies and online policies trained for downstream tasks, aiming to improve data efficiency and accelerate performance enhancement. An exi…

Reinforcement Learning (RL)

Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning

2021-06-09 · NeurIPS 2021 12 · Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong 외

Recent theoretical work studies sample-efficient reinforcement learning (RL) extensively in two settings: learning interactively in the environment (online RL), or learning from an offline dataset (offline RL). However, …

Offline RL Open-Ended Question Answering reinforcement-learning Reinforcement Learning+1

Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learning

2026-05-13 · Minung Kim, Jeongmo Kim, Gwanwoo Choi, Seungyul Han arxiv

Cross-domain offline reinforcement learning aims to adapt a policy from a source domain to a target domain using only pre-collected datasets, where environment dynamics may differ. A key challenge is to leverage source d…

Reinforcement Learning Offline RL

FOSP: Fine-tuning Offline Safe Policy through World Models

2024-07-06 · Chenyang Cao, Yucheng Xin, Silang Wu, Longxiang He 외

Offline Safe Reinforcement Learning (RL) seeks to address safety constraints by learning from static datasets and restricting exploration. However, these approaches heavily rely on the dataset and struggle to generalize …

Model-based Reinforcement Learning Offline RL Reinforcement Learning (RL)Safe Reinforcement Learning

Bridging Imitation and Online Reinforcement Learning: An Optimistic Tale

2023-03-20 · Botao Hao, Rahul Jain, Dengwang Tang, Zheng Wen

In this paper, we address the following problem: Given an offline demonstration dataset from an imperfect expert, what is the best way to leverage it to bootstrap online learning performance in MDPs. We first propose an …

Imitation Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)