Offline Policy Optimization with Variance Regularization
Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications. This is often because off-policy RL algorithms suffer from distributional shift, due to mismatch between dataset and the target policy, leading to high variance and over-estimation of value functions. In this work, we propose variance regularization for offline RL algorithms, using stationary distribution corrections. We show that by using Fenchel duality, we can avoid double sampling issues for computing the gradient of the variance regularizer. The proposed algorithm for offline variance regularization can be used to augment any existing offline policy optimization algorithms. We show that the regularizer leads to a lower bound to the offline policy optimization objective, which can help avoid over-estimation errors, and explains the benefits of our approach across a range of continuous control domains when compared to existing algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlOffline RLReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Offline Policy Optimization in RL with Variance Regularizaton
Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications. This is often because off-policy RL algorithms suffer from distributiona…
continuous-controlContinuous ControlOffline RLReinforcement Learning (RL)Supported Policy Optimization for Offline Reinforcement Learning
Policy constraint methods to offline reinforcement learning (RL) typically utilize parameterization or regularization that constrains the policy to perform actions within the support set of the behavior policy. The elabo…
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success…
ManagementMuJoCoMulti-agent Reinforcement LearningReinforcement Learning (RL)+2Behavior Proximal Policy Optimization
Offline reinforcement learning (RL) is a challenging setting where existing off-policy actor-critic methods perform poorly due to the overestimation of out-of-distribution state-action pairs. Thus, various additional aug…
D4RLOffline RLReinforcement Learning (RL)Offline Reinforcement Learning with Adaptive Behavior Regularization
Offline reinforcement learning (RL) defines a sample-efficient learning paradigm, where a policy is learned from static and previously collected datasets without additional interaction with the environment. The major obs…
D4RLOffline RLreinforcement-learningReinforcement Learning+1