An Optimistic Perspective on Offline Deep Reinforcement Learning
Off-policy reinforcement learning (RL) using a fixed offline dataset of logged interactions is an important consideration in real world applications. This paper studies offline RL using the DQN replay dataset comprising the entire replay experience of a DQN agent on 60 Atari 2600 games. We demonstrate that recent off-policy deep RL algorithms, even when trained solely on this fixed dataset, outperform the fully trained DQN agent. To enhance generalization in the offline setting, we present Random Ensemble Mixture (REM), a robust Q-learning algorithm that enforces optimal Bellman consistency on random convex combinations of multiple Q-value estimates. Offline REM trained on the DQN replay dataset surpasses strong RL baselines. Ablation studies highlight the role of offline dataset size and diversity as well as the algorithm choice in our positive results. Overall, the results here present an optimistic view that robust RL algorithms trained on sufficiently large and diverse offline datasets can lead to high quality policies. The DQN replay dataset can serve as an offline RL benchmark and is open-sourced.
Code (1)
Tasks
Atari GamesDeep Reinforcement LearningDiversityDQN Replay DatasetOffline RLQ-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
An Optimistic Perspective on Offline Reinforcement Learning
Off-policy reinforcement learning (RL) using a fixed offline dataset of logged interactions is an important consideration in real world applications. This paper studies offline RL using the DQN replay dataset comprising …
Atari GamesDiversityDQN Replay DatasetOffline RL+4Bayesian Design Principles for Offline-to-Online Reinforcement Learning
Offline reinforcement learning (RL) is crucial for real-world applications where exploration can be costly or unsafe. However, offline learned policies are often suboptimal, and further online fine-tuning is required. In…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
Offline-to-online (O2O) reinforcement learning (RL) provides an effective means of leveraging an offline pre-trained policy as initialization to improve performance rapidly with limited online interactions. Recent studie…
Offline RLReinforcement Learning (RL)Optimistic Model Rollouts for Pessimistic Offline Policy Optimization
Model-based offline reinforcement learning (RL) has made remarkable progress, offering a promising avenue for improving generalization with synthetic model rollouts. Existing works primarily focus on incorporating pessim…
modelOffline RLReinforcement Learning (RL)Human-centric Dialog Training via Offline Reinforcement Learning
How can we train a dialog model to produce better conversations by learning from human feedback, without the risk of humans teaching it harmful chat behaviors? We start by hosting models online, and gather human feedback…
Language ModellingOffline RLreinforcement-learningReinforcement Learning+1