BRAC+: Going Deeper with Behavior Regularized Offline Reinforcement Learning
Online interactions with the environment to collect data samples for training a Reinforcement Learning agent is not always feasible due to economic and safety concerns. The goal of Offline Reinforcement Learning (RL) is to address this problem by learning effective policies using previously collected datasets. Standard off-policy RL algorithms are prone to overestimations of the values of out of distribution (less explored) actions and are hence unsuitable for Offline RL. Behavior regularization, which constraints the learned policy within the support set of the dataset, has been proposed to tackle the limitations of standard off-policy algorithms. In this paper, we improve the behavior regularized offline reinforcement learning and propose \emph{BRAC+}. We use an analytical upper bound on KL divergence as the behavior regularizor to reduce variance associated with sample based estimations. Additionally, we employ state-dependent Lagrange multipliers for the regularization term to avoid distributing KL divergence penalty across all states of the sampled batch. The proposed Lagrange multipliers allow more freedom of deviation to high probability (more explored) states leading to better rewards while simultaneously restricting low probability (least explored) states to prevent out of distribution actions. We also propose several practical enhancements to further improve the performance. On challenging locomotion offline RL benchmarks, BRAC+ matches the state-of-the-art approaches on single-modal datasets and outperforms them on multi-modal datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning
Online interactions with the environment to collect data samples for training a Reinforcement Learning (RL) agent is not always feasible due to economic and safety concerns. The goal of Offline Reinforcement Learning is …
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Behavior Regularized Offline Reinforcement Learning
In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to a fixed offline …
continuous-controlContinuous ControlOffline RLreinforcement-learning+2Density-Ratio Weighted Behavioral Cloning: Learning Control Policies from Corrupted Datasets
Offline reinforcement learning (RL) enables policy optimization from fixed datasets, making it suitable for safety-critical applications where online exploration is infeasible. However, these datasets are often contamina…
Reinforcement LearningContinuous ControlOffline RLModel-Based Offline Meta-Reinforcement Learning with Regularization
Existing offline reinforcement learning (RL) methods face a few major challenges, particularly the distributional shift between the learned policy and the behavior policy. Offline Meta-RL is emerging as a promising appro…
Meta Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
Offline reinforcement learning (RL) seeks to learn optimal policies from static datasets without further environment interaction. A key challenge is the distribution shift between the learned and behavior policies, leadi…
Reinforcement LearningOffline RL