Online Pre-Training for Offline-to-Online Reinforcement Learning
Offline-to-online reinforcement learning (RL) aims to integrate the complementary strengths of offline and online RL by pre-training an agent offline and subsequently fine-tuning it through online interactions. However, recent studies reveal that offline pre-trained agents often underperform during online fine-tuning due to inaccurate value estimation caused by distribution shift, with random initialization proving more effective in certain cases. In this work, we propose a novel method, Online Pre-Training for Offline-to-Online RL (OPT), explicitly designed to address the issue of inaccurate value estimation in offline pre-trained agents. OPT introduces a new learning phase, Online Pre-Training, which allows the training of a new value function tailored specifically for effective online fine-tuning. Implementation of OPT on TD3 and SPOT demonstrates an average 30% improvement in performance across a wide range of D4RL environments, including MuJoCo, Antmaze, and Adroit.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningSimilar Papers 제목 키워드 기반
Guiding Online Reinforcement Learning with Action-Free Offline Pretraining
Offline RL methods have been shown to reduce the need for environment interaction by training agents using offline collected episodes. However, these methods typically require action information to be logged during data …
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles
Offline reinforcement learning (RL) is a learning paradigm where an agent learns from a fixed dataset of experience. However, learning solely from a static dataset can limit the performance due to the lack of exploration…
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)SAMG: State-Action-Aware Offline-to-Online Reinforcement Learning with Offline Model Guidance
The offline-to-online (O2O) paradigm in reinforcement learning (RL) utilizes pre-trained models on offline datasets for subsequent online fine-tuning. However, conventional O2O RL algorithms typically require maintaining…
D4RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
Offline reinforcement learning (RL) allows for the training of competent agents from offline datasets without any interaction with the environment. Online finetuning of such offline models can further improve performance…
DiversityOffline RLreinforcement-learningReinforcement Learning (RL)Unsupervised-to-Online Reinforcement Learning
Offline-to-online reinforcement learning (RL), a framework that trains a policy with offline RL and then further fine-tunes it with online RL, has been considered a promising recipe for data-driven decision-making. While…
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)