MOORe: Model-based Offline-to-Online Reinforcement Learning
With the success of offline reinforcement learning (RL), offline trained RL policies have the potential to be further improved when deployed online. A smooth transfer of the policy matters in safe real-world deployment. Besides, fast adaptation of the policy plays a vital role in practical online performance improvement. To tackle these challenges, we propose a simple yet efficient algorithm, Model-based Offline-to-Online Reinforcement learning (MOORe), which employs a prioritized sampling scheme that can dynamically adjust the offline and online data for smooth and efficient online adaptation of the policy. We provide a theoretical foundation for our algorithms design. Experiment results on the D4RL benchmark show that our algorithm smoothly transfers from offline to online stages while enabling sample-efficient online adaption, and also significantly outperforms existing methods.
Code (0)
등록된 구현이 없습니다.
Tasks
D4RLmodelreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Guiding Online Reinforcement Learning with Action-Free Offline Pretraining
Offline RL methods have been shown to reduce the need for environment interaction by training agents using offline collected episodes. However, these methods typically require action information to be logged during data …
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Online Pre-Training for Offline-to-Online Reinforcement Learning
Offline-to-online reinforcement learning (RL) aims to integrate the complementary strengths of offline and online RL by pre-training an agent offline and subsequently fine-tuning it through online interactions. However, …
Reinforcement LearningSAMG: State-Action-Aware Offline-to-Online Reinforcement Learning with Offline Model Guidance
The offline-to-online (O2O) paradigm in reinforcement learning (RL) utilizes pre-trained models on offline datasets for subsequent online fine-tuning. However, conventional O2O RL algorithms typically require maintaining…
D4RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Offline-Online Reinforcement Learning for Linear Mixture MDPs
We study offline-online reinforcement learning in linear mixture Markov decision processes (MDPs) under environment shift. In the offline phase, data are collected by an unknown behavior policy and may come from a mismat…
Reinforcement LearningEfficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning
Offline-to-Online Reinforcement Learning (O2O-RL) leverages an offline, pre-trained policy to minimize costly online interactions. Although data-efficient, O2O-RL is susceptible to shifts between offline and online distr…
Reinforcement Learning