Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
Improving the sample efficiency of reinforcement learning algorithms requires effective exploration. Following the principle of $\textit{optimism in the face of uncertainty}$ (OFU), we train a separate exploration policy to maximize the approximate upper confidence bound of the critics in an off-policy actor-critic framework. However, this introduces extra differences between the replay buffer and the target policy regarding their stationary state-action distributions. To mitigate the off-policy-ness, we adapt the recently introduced DICE framework to learn a distribution correction ratio for off-policy RL training. In particular, we correct the training distribution for both policies and critics. Empirically, we evaluate our proposed method in several challenging continuous control tasks and show superior performance compared to state-of-the-art methods. We also conduct extensive ablation studies to demonstrate the effectiveness and rationality of the proposed method.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous Controlreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration
When safety is formulated as a limit of cumulative cost, safe reinforcement learning (RL) aims to learn policies that maximize return subject to the cost constraint in data collection and deployment. Off-policy safe RL m…
Reinforcement LearningAutonomous DrivingOVD-Explorer: Optimism Should Not Be the Sole Pursuit of Exploration in Noisy Environments
In reinforcement learning, the optimism in the face of uncertainty (OFU) is a mainstream principle for directing exploration towards less explored areas, characterized by higher uncertainty. However, in the presence of e…
continuous-controlContinuous ControlMuJoCoEfficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
Model-based reinforcement learning algorithms with probabilistic dynamical models are amongst the most data-efficient learning methods. This is often attributed to their ability to distinguish between epistemic and aleat…
Model-based Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learning
The multi-agent setting is intricate and unpredictable since the behaviors of multiple agents influence one another. To address this environmental uncertainty, distributional reinforcement learning algorithms that incorp…
Distributional Reinforcement LearningMulti-agent Reinforcement Learningquantile regressionreinforcement-learning+2Bag of Policies for Distributional Deep Exploration
Efficient exploration in complex environments remains a major challenge for reinforcement learning (RL). Compared to previous Thompson sampling-inspired mechanisms that enable temporally extended exploration, i.e., deep …
Atari GamesEfficient ExplorationReinforcement Learning (RL)Thompson Sampling