Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Data

2023-07-10 · NeurIPS 2023 11

In some applications of reinforcement learning, a dataset of pre-collected experience is already available but it is also possible to acquire some additional online data to help improve the quality of the policy. However, it may be preferable to gather additional data with a single, non-reactive exploration policy and avoid the engineering costs associated with switching policies. In this paper we propose an algorithm with provable guarantees that can leverage an offline dataset to design a single non-reactive policy for exploration. We theoretically analyze the algorithm and measure the quality of the final policy as a function of the local coverage of the original dataset and the amount of additional data collected.

📄 PDF Abstract BibTeX arXiv:2307.04354

Code (0)

등록된 구현이 없습니다.

Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning

2021-06-09 · NeurIPS 2021 12 · Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong 외

Recent theoretical work studies sample-efficient reinforcement learning (RL) extensively in two settings: learning interactively in the environment (online RL), or learning from an offline dataset (offline RL). However, …

Offline RL Open-Ended Question Answering reinforcement-learning Reinforcement Learning+1

PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning

2023-05-25 · Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu 외

Offline-to-online reinforcement learning (RL), by combining the benefits of offline pretraining and online finetuning, promises enhanced sample efficiency and policy performance. However, existing methods, effective as t…

Computational Efficiency reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions

2023-03-30 · Yicheng Luo, Jackie Kay, Edward Grefenstette, Marc Peter Deisenroth

Offline reinforcement learning (RL) allows for the training of competent agents from offline datasets without any interaction with the environment. Online finetuning of such offline models can further improve performance…

Diversity Offline RL reinforcement-learning Reinforcement Learning (RL)

A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning

2023-12-12 · Yinmin Zhang, Jie Liu, Chuming Li, Yazhe Niu 외

Offline-to-online Reinforcement Learning (O2O RL) aims to improve the performance of offline pretrained policy using only a few online samples. Built on offline RL algorithms, most O2O methods focus on the balance betwee…

MuJoCo Offline RL

On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning

2021-06-15 · Abbas Abdolmaleki, Sandy H. Huang, Giulia Vezzani, Bobak Shahriari 외

Many advances that have improved the robustness and efficiency of deep reinforcement learning (RL) algorithms can, in one way or another, be understood as introducing additional objectives or constraints in the policy op…

Deep Reinforcement Learning Mixture-of-Experts Offline RL reinforcement-learning+1